From: Christoph Hellwig hch@lst.de

The write_same and write_zeroes limits for SCSI are effectively the same, so the current code works just fine. But we plan to remove REQ_OP_WRITE_SAME support, so switch to querying the write zeroes limit for a zeroing WRITE SAME operation.

Link: https://lore.kernel.org/r/20220209082828.2629273-2-hch@lst.de Reviewed-by: Chaitanya Kulkarni kch@nvidia.com Signed-off-by: Christoph Hellwig hch@lst.de Signed-off-by: Martin K. Petersen martin.petersen@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- drivers/scsi/cxlflash/vlun.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/cxlflash/vlun.c b/drivers/scsi/cxlflash/vlun.c index f1406ac77b0d..ea0a08a1696a 100644 --- a/drivers/scsi/cxlflash/vlun.c +++ b/drivers/scsi/cxlflash/vlun.c @@ -430,8 +430,8 @@ static int write_same16(struct scsi_device *sdev, struct device *dev = &cfg->dev->dev; const u32 s = ilog2(sdev->sector_size) - 9; const u32 to = sdev->request_queue->rq_timeout; - const u32 ws_limit = blk_queue_get_max_sectors(sdev->request_queue, - REQ_OP_WRITE_SAME) >> s; + const u32 ws_limit = + sdev->request_queue->limits.max_write_zeroes_sectors >> s;

cmd_buf = kzalloc(CMD_BUFSIZE, GFP_KERNEL); scsi_cmd = kzalloc(MAX_COMMAND_SIZE, GFP_KERNEL);

From: John Garry john.g.garry@oracle.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

Add atomic write support, as follows: - add helper functions to get request_queue atomic write limits - report request_queue atomic write support limits to sysfs and update Doc - support to safely merge atomic writes - deal with splitting atomic writes - misc helper functions - add a per-request atomic write flag

New request_queue limits are added, as follows: - atomic_write_hw_max is set by the block driver and is the maximum length of an atomic write which the device may support. It is not necessarily a power-of-2. - atomic_write_max_sectors is derived from atomic_write_hw_max_sectors and max_hw_sectors. It is always a power-of-2. Atomic writes may be merged, and atomic_write_max_sectors would be the limit on a merged atomic write request size. This value is not capped at max_sectors, as the value in max_sectors can be controlled from userspace, and it would only cause trouble if userspace could limit atomic_write_unit_max_bytes and the other atomic write limits. - atomic_write_hw_unit_{min,max} are set by the block driver and are the min/max length of an atomic write unit which the device may support. They both must be a power-of-2. Typically atomic_write_hw_unit_max will hold the same value as atomic_write_hw_max. - atomic_write_unit_{min,max} are derived from atomic_write_hw_unit_{min,max}, max_hw_sectors, and block core limits. Both min and max values must be a power-of-2. - atomic_write_hw_boundary is set by the block driver. If non-zero, it indicates an LBA space boundary at which an atomic write straddles no longer is atomically executed by the disk. The value must be a power-of-2. Note that it would be acceptable to enforce a rule that atomic_write_hw_boundary_sectors is a multiple of atomic_write_hw_unit_max, but the resultant code would be more complicated.

All atomic writes limits are by default set 0 to indicate no atomic write support. Even though it is assumed by Linux that a logical block can always be atomically written, we ignore this as it is not of particular interest. Stacked devices are just not supported either for now.

An atomic write must always be submitted to the block driver as part of a single request. As such, only a single BIO must be submitted to the block layer for an atomic write. When a single atomic write BIO is submitted, it cannot be split. As such, atomic_write_unit_{max, min}_bytes are limited by the maximum guaranteed BIO size which will not be required to be split. This max size is calculated by request_queue max segments and the number of bvecs a BIO can fit, BIO_MAX_VECS. Currently we rely on userspace issuing a write with iovcnt=1 for pwritev2() - as such, we can rely on each segment containing PAGE_SIZE of data, apart from the first+last, which each can fit logical block size of data. The first+last will be LBS length/aligned as we rely on direct IO alignment rules also.

New sysfs files are added to report the following atomic write limits: - atomic_write_unit_max_bytes - same as atomic_write_unit_max_sectors in bytes - atomic_write_unit_min_bytes - same as atomic_write_unit_min_sectors in bytes - atomic_write_boundary_bytes - same as atomic_write_hw_boundary_sectors in bytes - atomic_write_max_bytes - same as atomic_write_max_sectors in bytes

Atomic writes may only be merged with other atomic writes and only under the following conditions: - total resultant request length <= atomic_write_max_bytes - the merged write does not straddle a boundary

Helper function bdev_can_atomic_write() is added to indicate whether atomic writes may be issued to a bdev. If a bdev is a partition, the partition start must be aligned with both atomic_write_unit_min_sectors and atomic_write_hw_boundary_sectors.

FSes will rely on the block layer to validate that an atomic write BIO submitted will be of valid size, so add blk_validate_atomic_write_op_size() for this purpose. Userspace expects an atomic write which is of invalid size to be rejected with -EINVAL, so add BLK_STS_INVAL for this. Also use BLK_STS_INVAL for when a BIO needs to be split, as this should mean an invalid size BIO.

Flag REQ_ATOMIC is used for indicating an atomic write.

Co-developed-by: Himanshu Madhani himanshu.madhani@oracle.com Signed-off-by: Himanshu Madhani himanshu.madhani@oracle.com Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- Documentation/ABI/testing/sysfs-block | 52 +++++++++++++++ block/blk-core.c | 21 ++++++ block/blk-merge.c | 95 ++++++++++++++++++++++++++- block/blk-settings.c | 94 ++++++++++++++++++++++++++ block/blk-sysfs.c | 33 ++++++++++ include/linux/blk_types.h | 7 ++ include/linux/blkdev.h | 65 ++++++++++++++++++ 7 files changed, 366 insertions(+), 1 deletion(-)

diff --git a/Documentation/ABI/testing/sysfs-block b/Documentation/ABI/testing/sysfs-block index e34cdeeeb9d4..6cee984819b3 100644 --- a/Documentation/ABI/testing/sysfs-block +++ b/Documentation/ABI/testing/sysfs-block @@ -97,6 +97,58 @@ Description: indicates how many bytes the beginning of the device is offset from the disk's natural alignment.

+What: /sys/block/<disk>/atomic_write_max_bytes +Date: February 2024 +Contact: Himanshu Madhani himanshu.madhani@oracle.com +Description: + [RO] This parameter specifies the maximum atomic write + size reported by the device. This parameter is relevant + for merging of writes, where a merged atomic write + operation must not exceed this number of bytes. + This parameter may be greater to the value in + atomic_write_unit_max_bytes as + atomic_write_unit_max_bytes will be rounded down to a + power-of-two and atomic_write_unit_max_bytes may also be + limited by some other queue limits, such as max_segments. + This parameter - along with atomic_write_unit_min_bytes + and atomic_write_unit_max_bytes - will not be larger than + max_hw_sectors_kb, but may be larger than max_sectors_kb. + + +What: /sys/block/<disk>/atomic_write_unit_min_bytes +Date: February 2024 +Contact: Himanshu Madhani himanshu.madhani@oracle.com +Description: + [RO] This parameter specifies the smallest block which can + be written atomically with an atomic write operation. All + atomic write operations must begin at a + atomic_write_unit_min boundary and must be multiples of + atomic_write_unit_min. This value must be a power-of-two. + + +What: /sys/block/<disk>/atomic_write_unit_max_bytes +Date: February 2024 +Contact: Himanshu Madhani himanshu.madhani@oracle.com +Description: + [RO] This parameter defines the largest block which can be + written atomically with an atomic write operation. This + value must be a multiple of atomic_write_unit_min and must + be a power-of-two. This value will not be larger than + atomic_write_max_bytes. + + +What: /sys/block/<disk>/atomic_write_boundary_bytes +Date: February 2024 +Contact: Himanshu Madhani himanshu.madhani@oracle.com +Description: + [RO] A device may need to internally split I/Os which + straddle a given logical block address boundary. In that + case a single atomic write operation will be processed as + one of more sub-operations which each complete atomically. + This parameter specifies the size in bytes of the atomic + boundary if one is reported by the device. This value must + be a power-of-two. + What: /sys/block/<disk>/<partition>/alignment_offset Date: April 2009 Contact: Martin K. Petersen martin.petersen@oracle.com diff --git a/block/blk-core.c b/block/blk-core.c index 7950063c0345..27f356c07d26 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -205,6 +205,8 @@ static const struct { [BLK_STS_ZONE_OPEN_RESOURCE] = { -ETOOMANYREFS, "open zones exceeded" }, [BLK_STS_ZONE_ACTIVE_RESOURCE] = { -EOVERFLOW, "active zones exceeded" },

+ [BLK_STS_INVAL] = { -EINVAL, "invalid" }, + /* everything else not covered above: */ [BLK_STS_IOERR] = { -EIO, "I/O" }, }; @@ -819,6 +821,18 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q, return BLK_STS_OK; }

+static blk_status_t blk_validate_atomic_write_op_size(struct request_queue *q, + struct bio *bio) +{ + if (bio->bi_iter.bi_size > queue_atomic_write_unit_max_bytes(q)) + return BLK_STS_INVAL; + + if (bio->bi_iter.bi_size % queue_atomic_write_unit_min_bytes(q)) + return BLK_STS_INVAL; + + return BLK_STS_OK; +} + static noinline_for_stack bool submit_bio_checks(struct bio *bio) { struct request_queue *q = bio->bi_disk->queue; @@ -900,6 +914,13 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) if (!q->limits.max_write_zeroes_sectors) goto not_supported; break; + case REQ_OP_WRITE: + if (bio->bi_opf & REQ_ATOMIC) { + status = blk_validate_atomic_write_op_size(q, bio); + if (status != BLK_STS_OK) + goto end_io; + } + break; default: break; } diff --git a/block/blk-merge.c b/block/blk-merge.c index a65d1d275040..7ca680a6c037 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -13,6 +13,46 @@ #include "blk.h" #include "blk-rq-qos.h"

+/* + * rq_straddles_atomic_write_boundary - check for boundary violation + * @rq: request to check + * @front: data size to be appended to front + * @back: data size to be appended to back + * + * Determine whether merging a request or bio into another request will result + * in a merged request which straddles an atomic write boundary. + * + * The value @front_adjust is the data which would be appended to the front of + * @rq, while the value @back_adjust is the data which would be appended to the + * back of @rq. Callers will typically only have either @front_adjust or + * @back_adjust as non-zero. + * + */ +static bool rq_straddles_atomic_write_boundary(struct request *rq, + unsigned int front_adjust, + unsigned int back_adjust) +{ + unsigned int boundary = queue_atomic_write_boundary_bytes(rq->q); + u64 mask, start_rq_pos, end_rq_pos; + + if (!boundary) + return false; + + start_rq_pos = blk_rq_pos(rq) << SECTOR_SHIFT; + end_rq_pos = start_rq_pos + blk_rq_bytes(rq) - 1; + + start_rq_pos -= front_adjust; + end_rq_pos += back_adjust; + + mask = ~(boundary - 1); + + /* Top bits are different, so crossed a boundary */ + if ((start_rq_pos & mask) != (end_rq_pos & mask)) + return true; + + return false; +} + static inline bool bio_will_gap(struct request_queue *q, struct request *prev_rq, struct bio *prev, struct bio *next) { @@ -145,11 +185,20 @@ static inline unsigned get_max_io_size(struct request_queue *q, struct bio *bio) { unsigned sectors = blk_max_size_offset(q, bio->bi_iter.bi_sector, 0); - unsigned max_sectors = sectors; + unsigned max_sectors; unsigned pbs = queue_physical_block_size(q) >> SECTOR_SHIFT; unsigned lbs = queue_logical_block_size(q) >> SECTOR_SHIFT; unsigned start_offset = bio->bi_iter.bi_sector & (pbs - 1);

+ /* + * We ignore lim->max_sectors for atomic writes simply because + * it may less than the bio size, which we cannot tolerate. + */ + if (bio->bi_opf & REQ_ATOMIC) + max_sectors = q->limits.atomic_write_max_sectors; + else + max_sectors = sectors; + max_sectors += start_offset; max_sectors &= ~(pbs - 1); if (max_sectors > start_offset) @@ -278,6 +327,11 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, *segs = nsegs; return NULL; split: + if (bio->bi_opf & REQ_ATOMIC) { + bio->bi_status = BLK_STS_INVAL; + bio_endio(bio); + return ERR_PTR(-EINVAL); + } *segs = nsegs; return bio_split(bio, sectors, GFP_NOIO, bs); } @@ -594,6 +648,13 @@ int ll_back_merge_fn(struct request *req, struct bio *bio, unsigned int nr_segs) return 0; }

+ if (req->cmd_flags & REQ_ATOMIC) { + if (rq_straddles_atomic_write_boundary(req, + bio->bi_iter.bi_size, 0)) { + return 0; + } + } + return ll_new_hw_segment(req, bio, nr_segs); }

@@ -613,6 +674,13 @@ static int ll_front_merge_fn(struct request *req, struct bio *bio, return 0; }

+ if (req->cmd_flags & REQ_ATOMIC) { + if (rq_straddles_atomic_write_boundary(req, + 0, bio->bi_iter.bi_size)) { + return 0; + } + } + return ll_new_hw_segment(req, bio, nr_segs); }

@@ -649,6 +717,13 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req, blk_rq_get_max_sectors(req, blk_rq_pos(req))) return 0;

+ if (req->cmd_flags & REQ_ATOMIC) { + if (rq_straddles_atomic_write_boundary(req, + 0, blk_rq_bytes(next))) { + return 0; + } + } + total_phys_segments = req->nr_phys_segments + next->nr_phys_segments; if (total_phys_segments > blk_rq_get_max_segments(req)) return 0; @@ -721,6 +796,18 @@ static enum elv_merge blk_try_req_merge(struct request *req, return ELEVATOR_NO_MERGE; }

+static bool blk_atomic_write_mergeable_rq_bio(struct request *rq, + struct bio *bio) +{ + return (rq->cmd_flags & REQ_ATOMIC) == (bio->bi_opf & REQ_ATOMIC); +} + +static bool blk_atomic_write_mergeable_rqs(struct request *rq, + struct request *next) +{ + return (rq->cmd_flags & REQ_ATOMIC) == (next->cmd_flags & REQ_ATOMIC); +} + /* * For non-mq, this has to be called with the request spinlock acquired. * For mq with scheduling, the appropriate queue wide lock should be held. @@ -752,6 +839,9 @@ static struct request *attempt_merge(struct request_queue *q, if (req->ioprio != next->ioprio) return NULL;

+ if (!blk_atomic_write_mergeable_rqs(req, next)) + return NULL; + /* * If we are allowed to merge, then append bio list * from next to rq and release next. merge_requests_fn @@ -895,6 +985,9 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (rq->ioprio != bio_prio(bio)) return false;

+ if (blk_atomic_write_mergeable_rq_bio(rq, bio) == false) + return false; + return true; }

diff --git a/block/blk-settings.c b/block/blk-settings.c index c3aa7f8ee388..754b758946a9 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -127,6 +127,41 @@ void blk_queue_bounce_limit(struct request_queue *q, u64 max_addr) } EXPORT_SYMBOL(blk_queue_bounce_limit);

+/* + * Returns max guaranteed bytes which we can fit in a bio. + * + * We always assume that we can fit in at least PAGE_SIZE in a segment, apart + * from first and last segments. + */ +static +unsigned int blk_queue_max_guaranteed_bio(struct queue_limits *limits) +{ + unsigned int max_segments = min((u16)BIO_MAX_PAGES, limits->max_segments); + unsigned int length; + + length = min(max_segments, 2U) * limits->logical_block_size; + if (max_segments > 2) + length += (max_segments - 2) * PAGE_SIZE; + + return length; +} + +static void blk_atomic_writes_update_limits(struct queue_limits *limits) +{ + unsigned int unit_limit = min(limits->max_hw_sectors << SECTOR_SHIFT, + blk_queue_max_guaranteed_bio(limits)); + + unit_limit = rounddown_pow_of_two(unit_limit); + + limits->atomic_write_max_sectors = + min(limits->atomic_write_hw_max >> SECTOR_SHIFT, + limits->max_hw_sectors); + limits->atomic_write_unit_min = + min(limits->atomic_write_hw_unit_min, unit_limit); + limits->atomic_write_unit_max = + min(limits->atomic_write_hw_unit_max, unit_limit); +} + /** * blk_queue_max_hw_sectors - set max sectors for a request for this queue * @q: the request queue for the device @@ -161,6 +196,9 @@ void blk_queue_max_hw_sectors(struct request_queue *q, unsigned int max_hw_secto max_sectors = min_not_zero(max_hw_sectors, limits->max_dev_sectors); max_sectors = min_t(unsigned int, max_sectors, BLK_DEF_MAX_SECTORS); limits->max_sectors = max_sectors; + + blk_atomic_writes_update_limits(limits); + q->backing_dev_info->io_pages = max_sectors >> (PAGE_SHIFT - 9); } EXPORT_SYMBOL(blk_queue_max_hw_sectors); @@ -196,6 +234,62 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors);

+/** + * blk_queue_atomic_write_max_bytes - set max bytes supported by + * the device for atomic write operations. + * @q: the request queue for the device + * @bytes: maximum bytes supported + */ +void blk_queue_atomic_write_max_bytes(struct request_queue *q, + unsigned int bytes) +{ + q->limits.atomic_write_hw_max = bytes; + blk_atomic_writes_update_limits(&q->limits); +} +EXPORT_SYMBOL(blk_queue_atomic_write_max_bytes); + +/** + * blk_queue_atomic_write_boundary_bytes - Device's logical block address space + * which an atomic write should not cross. + * @q: the request queue for the device + * @bytes: must be a power-of-two. + */ +void blk_queue_atomic_write_boundary_bytes(struct request_queue *q, + unsigned int bytes) +{ + q->limits.atomic_write_hw_boundary = bytes; +} +EXPORT_SYMBOL(blk_queue_atomic_write_boundary_bytes); + +/** + * blk_queue_atomic_write_unit_min_bytes - smallest unit that can be written + * atomically to the device. + * @q: the request queue for the device + * @bytes: must be a power-of-two. + */ +void blk_queue_atomic_write_unit_min_bytes(struct request_queue *q, + unsigned int bytes) +{ + q->limits.atomic_write_hw_unit_min = bytes; + blk_atomic_writes_update_limits(&q->limits); +} +EXPORT_SYMBOL(blk_queue_atomic_write_unit_min_bytes); + +/* + * blk_queue_atomic_write_unit_max_bytes - largest unit that can be written + * atomically to the device. + * @q: the request queue for the device + * @bytes: must be a power-of-two. + */ +void blk_queue_atomic_write_unit_max_bytes(struct request_queue *q, + unsigned int bytes) +{ + q->limits.atomic_write_hw_unit_max = bytes; + blk_atomic_writes_update_limits(&q->limits); +} +EXPORT_SYMBOL(blk_queue_atomic_write_unit_max_bytes); + + /** * blk_queue_max_write_same_sectors - set max sectors for a single write same * @q: the request queue for the device diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index c95be9626a09..6916f1ed4590 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -126,6 +126,30 @@ static ssize_t queue_max_discard_segments_show(struct request_queue *q, return queue_var_show(queue_max_discard_segments(q), (page)); }

+static ssize_t queue_atomic_write_max_bytes_show(struct request_queue *q, + char *page) +{ + return queue_var_show(queue_atomic_write_max_bytes(q), page); +} + +static ssize_t queue_atomic_write_boundary_show(struct request_queue *q, + char *page) +{ + return queue_var_show(queue_atomic_write_boundary_bytes(q), page); +} + +static ssize_t queue_atomic_write_unit_min_show(struct request_queue *q, + char *page) +{ + return queue_var_show(queue_atomic_write_unit_min_bytes(q), page); +} + +static ssize_t queue_atomic_write_unit_max_show(struct request_queue *q, + char *page) +{ + return queue_var_show(queue_atomic_write_unit_max_bytes(q), page); +} + static ssize_t queue_max_integrity_segments_show(struct request_queue *q, char *page) { return queue_var_show(q->limits.max_integrity_segments, (page)); @@ -585,6 +609,11 @@ QUEUE_RO_ENTRY(queue_discard_max_hw, "discard_max_hw_bytes"); QUEUE_RW_ENTRY(queue_discard_max, "discard_max_bytes"); QUEUE_RO_ENTRY(queue_discard_zeroes_data, "discard_zeroes_data");

+QUEUE_RO_ENTRY(queue_atomic_write_max_bytes, "atomic_write_max_bytes"); +QUEUE_RO_ENTRY(queue_atomic_write_boundary, "atomic_write_boundary_bytes"); +QUEUE_RO_ENTRY(queue_atomic_write_unit_max, "atomic_write_unit_max_bytes"); +QUEUE_RO_ENTRY(queue_atomic_write_unit_min, "atomic_write_unit_min_bytes"); + QUEUE_RO_ENTRY(queue_write_same_max, "write_same_max_bytes"); QUEUE_RO_ENTRY(queue_write_zeroes_max, "write_zeroes_max_bytes"); QUEUE_RO_ENTRY(queue_zone_append_max, "zone_append_max_bytes"); @@ -639,6 +668,10 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_atomic_write_max_bytes_entry.attr, + &queue_atomic_write_boundary_entry.attr, + &queue_atomic_write_unit_min_entry.attr, + &queue_atomic_write_unit_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 1853ec569b72..996e03d50f41 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -133,6 +133,11 @@ typedef u8 __bitwise blk_status_t; */ #define BLK_STS_ZONE_ACTIVE_RESOURCE ((__force blk_status_t)16)

+/* + * Invalid size or alignment. + */ +#define BLK_STS_INVAL ((__force blk_status_t)19) + /** * blk_path_error - returns true if error may be path related * @error: status the request was completed with @@ -421,6 +426,7 @@ enum req_flag_bits { /* for driver use */ __REQ_DRV, __REQ_SWAP, /* swapping request. */ + __REQ_ATOMIC, /* for atomic write operations */ __REQ_NR_BITS, /* stops here */ };

@@ -445,6 +451,7 @@ enum req_flag_bits {

#define REQ_DRV (1ULL << __REQ_DRV) #define REQ_SWAP (1ULL << __REQ_SWAP) +#define REQ_ATOMIC (1ULL << __REQ_ATOMIC)

#define REQ_FAILFAST_MASK \ (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 2d6747877f7c..a53e70a6aeec 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -346,6 +346,16 @@ struct queue_limits { unsigned int discard_granularity; unsigned int discard_alignment;

+ /* atomic write limits */ + unsigned int atomic_write_hw_max; + unsigned int atomic_write_max_sectors; + unsigned int atomic_write_hw_boundary; + unsigned int atomic_write_hw_unit_min; + unsigned int atomic_write_unit_min; + unsigned int atomic_write_hw_unit_max; + unsigned int atomic_write_unit_max; + + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -1097,6 +1107,9 @@ static inline unsigned int blk_queue_get_max_sectors(struct request *rq) if (unlikely(op == REQ_OP_WRITE_ZEROES)) return q->limits.max_write_zeroes_sectors;

+ if (rq->cmd_flags & REQ_ATOMIC) + return q->limits.atomic_write_max_sectors; + return q->limits.max_sectors; }

@@ -1187,6 +1200,14 @@ extern void blk_queue_max_zone_append_sectors(struct request_queue *q, extern void blk_queue_physical_block_size(struct request_queue *, unsigned int); extern void blk_queue_alignment_offset(struct request_queue *q, unsigned int alignment); +void blk_queue_atomic_write_max_bytes(struct request_queue *q, + unsigned int bytes); +void blk_queue_atomic_write_boundary_bytes(struct request_queue *q, + unsigned int bytes); +void blk_queue_atomic_write_unit_max_bytes(struct request_queue *q, + unsigned int bytes); +void blk_queue_atomic_write_unit_min_bytes(struct request_queue *q, + unsigned int bytes); void blk_queue_update_readahead(struct request_queue *q); extern void blk_limits_io_min(struct queue_limits *limits, unsigned int min); extern void blk_queue_io_min(struct request_queue *q, unsigned int min); @@ -1649,6 +1670,30 @@ static inline unsigned int bdev_max_active_zones(struct block_device *bdev) return 0; }

+static inline unsigned int +queue_atomic_write_unit_max_bytes(const struct request_queue *q) +{ + return q->limits.atomic_write_unit_max; +} + +static inline unsigned int +queue_atomic_write_unit_min_bytes(const struct request_queue *q) +{ + return q->limits.atomic_write_unit_min; +} + +static inline unsigned int +queue_atomic_write_boundary_bytes(const struct request_queue *q) +{ + return q->limits.atomic_write_hw_boundary; +} + +static inline unsigned int +queue_atomic_write_max_bytes(const struct request_queue *q) +{ + return q->limits.atomic_write_max_sectors << SECTOR_SHIFT; +} + static inline int queue_dma_alignment(const struct request_queue *q) { return q ? q->dma_alignment : 511; @@ -2102,4 +2147,24 @@ int fsync_bdev(struct block_device *bdev); struct super_block *freeze_bdev(struct block_device *bdev); int thaw_bdev(struct block_device *bdev, struct super_block *sb);

+static inline bool bdev_can_atomic_write(struct block_device *bdev) +{ + struct request_queue *bd_queue = bdev_get_queue(bdev); + struct queue_limits *limits = &bd_queue->limits; + + if (!limits->atomic_write_unit_min) + return false; + + if (bdev_is_partition(bdev)) { + sector_t bd_start_sect = bdev->bd_part->start_sect; + unsigned int alignment = + max(limits->atomic_write_unit_min, + limits->atomic_write_hw_boundary); + if (!IS_ALIGNED(bd_start_sect, alignment)) + return false; + } + + return true; +} + #endif /* _LINUX_BLKDEV_H */

From: Alan Adamson alan.adamson@oracle.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

Add support to set block layer request_queue atomic write limits. The limits will be derived from either the namespace or controller atomic parameters.

NVMe atomic-related parameters are grouped into "normal" and "power-fail" (or PF) class of parameter. For atomic write support, only PF parameters are of interest. The "normal" parameters are concerned with racing reads and writes (which also applies to PF). See NVM Command Set Specification Revision 1.0d section 2.1.4 for reference.

Whether to use per namespace or controller atomic parameters is decided by NSFEAT bit 1 - see Figure 97: Identify – Identify Namespace Data Structure, NVM Command Set.

NVMe namespaces may define an atomic boundary, whereby no atomic guarantees are provided for a write which straddles this per-lba space boundary. The block layer merging policy is such that no merges may occur in which the resultant request would straddle such a boundary.

Unlike SCSI, NVMe specifies no granularity or alignment rules, apart from atomic boundary rule. In addition, again unlike SCSI, there is no dedicated atomic write command - a write which adheres to the atomic size limit and boundary is implicitly atomic.

If NSFEAT bit 1 is set, the following parameters are of interest: - NAWUPF (Namespace Atomic Write Unit Power Fail) - NABSPF (Namespace Atomic Boundary Size Power Fail) - NABO (Namespace Atomic Boundary Offset)

and we set request_queue limits as follows: - atomic_write_unit_max = rounddown_pow_of_two(NAWUPF) - atomic_write_max_bytes = NAWUPF - atomic_write_boundary = NABSPF

If in the unlikely scenario that NABO is non-zero, then atomic writes will not be supported at all as dealing with this adds extra complexity. This policy may change in future.

In all cases, atomic_write_unit_min is set to the logical block size.

If NSFEAT bit 1 is unset, the following parameter is of interest: - AWUPF (Atomic Write Unit Power Fail)

and we set request_queue limits as follows: - atomic_write_unit_max = rounddown_pow_of_two(AWUPF) - atomic_write_max_bytes = AWUPF - atomic_write_boundary = 0

A new function, nvme_valid_atomic_write(), is also called from submission path to verify that a request has been submitted to the driver will actually be executed atomically. As mentioned, there is no dedicated NVMe atomic write command (which may error for a command which exceeds the controller atomic write limits).

Note on NABSPF: There seems to be some vagueness in the spec as to whether NABSPF applies for NSFEAT bit 1 being unset. Figure 97 does not explicitly mention NABSPF and how it is affected by bit 1. However Figure 4 does tell to check Figure 97 for info about per-namespace parameters, which NABSPF is, so it is implied. However currently nvme_update_disk_info() does check namespace parameter NABO regardless of this bit.

Signed-off-by: Alan Adamson alan.adamson@oracle.com Reviewed-by: Keith Busch kbusch@kernel.org jpg: total rewrite Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- drivers/nvme/host/core.c | 52 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 9fcc05c4f88c..4fa0c55787d2 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -752,6 +752,30 @@ static inline blk_status_t nvme_setup_write_zeroes(struct nvme_ns *ns, return BLK_STS_OK; }

+static bool nvme_valid_atomic_write(struct request *req) +{ + struct request_queue *q = req->q; + u32 boundary_bytes = queue_atomic_write_boundary_bytes(q); + + if (blk_rq_bytes(req) > queue_atomic_write_unit_max_bytes(q)) + return false; + + if (boundary_bytes) { + u64 mask = boundary_bytes - 1, imask = ~mask; + u64 start = blk_rq_pos(req) << SECTOR_SHIFT; + u64 end = start + blk_rq_bytes(req) - 1; + + /* If greater then must be crossing a boundary */ + if (blk_rq_bytes(req) > boundary_bytes) + return false; + + if ((start & imask) != (end & imask)) + return false; + } + + return true; +} + static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd, enum nvme_opcode op) @@ -768,6 +792,13 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, if (req->cmd_flags & REQ_RAHEAD) dsmgmt |= NVME_RW_DSM_FREQ_PREFETCH;

+ /* + * Ensure that nothing has been sent which cannot be executed + * atomically. + */ + if (req->cmd_flags & REQ_ATOMIC && !nvme_valid_atomic_write(req)) + return BLK_STS_INVAL; + cmnd->rw.opcode = op; cmnd->rw.nsid = cpu_to_le32(ns->head->ns_id); cmnd->rw.slba = cpu_to_le64(nvme_sect_to_lba(ns, blk_rq_pos(req))); @@ -2011,6 +2042,22 @@ static int nvme_configure_metadata(struct nvme_ns *ns, struct nvme_id_ns *id) return 0; }

+static void nvme_update_atomic_write_disk_info(struct nvme_ns *ns, + struct nvme_id_ns *id, struct queue_limits *lim, + u32 bs, u32 atomic_bs) +{ + unsigned int boundary = 0; + + if (id->nsfeat & NVME_NS_FEAT_ATOMICS && id->nawupf) { + if (le16_to_cpu(id->nabspf)) + boundary = (le16_to_cpu(id->nabspf) + 1) * bs; + } + lim->atomic_write_hw_max = atomic_bs; + lim->atomic_write_hw_boundary = boundary; + lim->atomic_write_hw_unit_min = bs; + lim->atomic_write_hw_unit_max = rounddown_pow_of_two(atomic_bs); +} + static void nvme_set_queue_limits(struct nvme_ctrl *ctrl, struct request_queue *q) { @@ -2060,6 +2107,9 @@ static void nvme_update_disk_info(struct gendisk *disk, atomic_bs = (1 + le16_to_cpu(id->nawupf)) * bs; else atomic_bs = (1 + ns->ctrl->subsys->awupf) * bs; + + nvme_update_atomic_write_disk_info(ns, id, &disk->queue->limits, + bs, atomic_bs); }

if (id->nsfeat & NVME_NS_FEAT_IO_OPT) { @@ -2160,6 +2210,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id) goto out_unfreeze; nvme_set_chunk_sectors(ns, id); nvme_update_disk_info(ns->disk, ns, id); + nvme_set_queue_limits(ns->ctrl, ns->queue); blk_mq_unfreeze_queue(ns->disk->queue);

if (blk_queue_is_zoned(ns->queue)) { @@ -2172,6 +2223,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_id_ns *id) if (ns->head->disk) { blk_mq_freeze_queue(ns->head->disk->queue); nvme_update_disk_info(ns->head->disk, ns, id); + nvme_set_queue_limits(ns->ctrl, ns->head->disk->queue); blk_stack_limits(&ns->head->disk->queue->limits, &ns->queue->limits, 0); blk_queue_update_readahead(ns->head->disk->queue);

From: "Darrick J. Wong" djwong@kernel.org

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

Enable this feature.

Signed-off-by: "Darrick J. Wong" djwong@kernel.org Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/libxfs/xfs_format.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/xfs/libxfs/xfs_format.h b/fs/xfs/libxfs/xfs_format.h index fb537bb7dba5..71629d6dfd1b 100644 --- a/fs/xfs/libxfs/xfs_format.h +++ b/fs/xfs/libxfs/xfs_format.h @@ -358,7 +358,8 @@ xfs_sb_has_compat_feature( (XFS_SB_FEAT_RO_COMPAT_FINOBT | \ XFS_SB_FEAT_RO_COMPAT_RMAPBT | \ XFS_SB_FEAT_RO_COMPAT_REFLINK| \ - XFS_SB_FEAT_RO_COMPAT_INOBTCNT) + XFS_SB_FEAT_RO_COMPAT_INOBTCNT | \ + XFS_SB_FEAT_RO_COMPAT_FORCEALIGN) #define XFS_SB_FEAT_RO_COMPAT_UNKNOWN ~XFS_SB_FEAT_RO_COMPAT_ALL static inline bool xfs_sb_has_ro_compat_feature(

From: John Garry john.g.garry@oracle.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

For when forcealign is enabled, we want the EOF to be aligned as well, so do not free EOF blocks.

Add helper function xfs_get_extsz() to get the guaranteed extent size alignment for forcealign enabled. Since this code is only relevant to forcealign and forcealign is not possible for RT yet, ignore RT.

Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/xfs_bmap_util.c | 3 +++ fs/xfs/xfs_inode.c | 14 ++++++++++++++ fs/xfs/xfs_inode.h | 1 + 3 files changed, 18 insertions(+)

diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c index edf62092125c..94e5d57de432 100644 --- a/fs/xfs/xfs_bmap_util.c +++ b/fs/xfs/xfs_bmap_util.c @@ -654,6 +654,9 @@ xfs_free_eofblocks( * of the file. If not, then there is nothing to do. */ end_fsb = XFS_B_TO_FSB(mp, (xfs_ufsize_t)XFS_ISIZE(ip)); + /* Do not free blocks when forcing extent sizes */ + if (xfs_get_extsz(ip) > 1) + end_fsb = roundup_64(end_fsb, xfs_get_extsz(ip)); last_fsb = XFS_B_TO_FSB(mp, mp->m_super->s_maxbytes); if (last_fsb <= end_fsb) return 0; diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index d858f0b1ca02..ae4282eb058d 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -68,6 +68,20 @@ xfs_get_extsz_hint( return 0; }

+/* + * Helper function to extract extent size. It will return a power-of-2, + * as forcealign requires this. + */ +xfs_extlen_t +xfs_get_extsz( + struct xfs_inode *ip) +{ + if (xfs_inode_forcealign(ip) && ip->i_d.di_extsize) + return ip->i_d.di_extsize; + + return 1; +} + /* * Helper function to extract CoW extent size hint from inode. * Between the extent size hint and the CoW extent size hint, we diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h index 53a2b90486f6..5d740f793782 100644 --- a/fs/xfs/xfs_inode.h +++ b/fs/xfs/xfs_inode.h @@ -493,6 +493,7 @@ void xfs_lock_two_inodes(struct xfs_inode *ip0, uint ip0_mode, struct xfs_inode *ip1, uint ip1_mode);

xfs_extlen_t xfs_get_extsz_hint(struct xfs_inode *ip); +xfs_extlen_t xfs_get_extsz(struct xfs_inode *ip); xfs_extlen_t xfs_get_cowextsz_hint(struct xfs_inode *ip);

int xfs_dir_ialloc(struct xfs_trans **, struct xfs_inode *, umode_t,

From: John Garry john.g.garry@oracle.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

Set iomap->extent_shift when sub-extent zeroing is required.

We treat a sub-extent write same as an unaligned write, so we can leverage the existing sub-FSblock unaligned write support, i.e. try a shared lock with IOMAP_DIO_OVERWRITE_ONLY flag, if this fails then try the exclusive lock.

In xfs_iomap_write_unwritten(), FSB calcs are now based on the extsize.

Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/xfs_file.c | 4 ++-- fs/xfs/xfs_iomap.c | 15 +++++++++++++-- 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 52643eac5d46..1e4da3479432 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -597,8 +597,8 @@ xfs_file_dio_aio_write( * the inode as necessary for EOF zeroing cases and fill out the new * inode size as appropriate. */ - if ((iocb->ki_pos & mp->m_blockmask) || - ((iocb->ki_pos + count) & mp->m_blockmask)) { + if ((iocb->ki_pos & (XFS_FSB_TO_B(mp, xfs_get_extsz(ip)) - 1)) || + ((iocb->ki_pos + count) & (XFS_FSB_TO_B(mp, xfs_get_extsz(ip)) - 1))) { unaligned_io = 1;

/* diff --git a/fs/xfs/xfs_iomap.c b/fs/xfs/xfs_iomap.c index af2fe2e2bd4b..dc01689988ab 100644 --- a/fs/xfs/xfs_iomap.c +++ b/fs/xfs/xfs_iomap.c @@ -90,6 +90,7 @@ xfs_bmbt_to_iomap( { struct xfs_mount *mp = ip->i_mount; struct xfs_buftarg *target = xfs_inode_buftarg(ip); + xfs_extlen_t extsz = xfs_get_extsz(ip);

if (unlikely(!xfs_valid_startblock(ip, imap->br_startblock))) return xfs_alert_fsblock_zero(ip, imap); @@ -120,6 +121,8 @@ xfs_bmbt_to_iomap(

iomap->validity_cookie = sequence_cookie; iomap->page_ops = &xfs_iomap_page_ops; + if (extsz > 1) + iomap->extent_shift = ffs(extsz) - 1; return 0; }

@@ -546,11 +549,19 @@ xfs_iomap_write_unwritten( xfs_fsize_t i_size; uint resblks; int error; + xfs_extlen_t extsz = xfs_get_extsz(ip);

trace_xfs_unwritten_convert(ip, offset, count);

- offset_fsb = XFS_B_TO_FSBT(mp, offset); - count_fsb = XFS_B_TO_FSB(mp, (xfs_ufsize_t)offset + count); + if (extsz > 1) { + xfs_extlen_t extsize_bytes = XFS_FSB_TO_B(mp, extsz); + + offset_fsb = XFS_B_TO_FSBT(mp, round_down(offset, extsize_bytes)); + count_fsb = XFS_B_TO_FSB(mp, round_up(offset + count, extsize_bytes)); + } else { + offset_fsb = XFS_B_TO_FSBT(mp, offset); + count_fsb = XFS_B_TO_FSB(mp, (xfs_ufsize_t)offset + count); + } count_fsb = (xfs_filblks_t)(count_fsb - offset_fsb);

/*

From: John Garry john.g.garry@oracle.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

Support atomic writes by producing a single BIO with REQ_ATOMIC flag set.

We rely on the FS to guarantee extent alignment, such that an atomic write should never straddle two or more extents. The FS should also check for validity of an atomic write length/alignment.

Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- fs/iomap/direct-io.c | 10 ++++++++++ 1 file changed, 10 insertions(+)

diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 6cdacdd141f6..063ea9eb8afb 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -234,6 +234,7 @@ static loff_t iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, struct iomap_dio *dio, struct iomap *iomap) { + bool is_atomic = dio->iocb->ki_flags & IOCB_ATOMIC; unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev)); unsigned int zeroing_size, pad; unsigned int align = iov_iter_alignment(dio->submit.iter); @@ -307,6 +308,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, bio->bi_iter.bi_sector = iomap_sector(iomap, pos); bio->bi_write_hint = dio->iocb->ki_hint; bio->bi_ioprio = dio->iocb->ki_ioprio; + bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io;

@@ -323,8 +325,16 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, }

n = bio->bi_iter.bi_size; + if (is_atomic && (n != orig_count)) { + /* This bio should have covered the complete length */ + ret = -EINVAL; + bio_put(bio); + goto out; + } if (dio->flags & IOMAP_DIO_WRITE) { bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE; + if (is_atomic) + bio->bi_opf |= REQ_ATOMIC; if (use_fua) bio->bi_opf |= REQ_FUA; else

From: John Garry john.g.garry@oracle.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

Validate that an atomic write adheres to length/offset rules. Since we require extent alignment for atomic writes, this effectively also enforces that the BIO which iomap produces is aligned.

Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/xfs_file.c | 8 ++++++++ 1 file changed, 8 insertions(+)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 1e4da3479432..75d7ca8d2a4a 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -586,6 +586,14 @@ xfs_file_dio_aio_write( size_t count = iov_iter_count(from); struct xfs_buftarg *target = xfs_inode_buftarg(ip);

+ if (iocb->ki_flags & IOCB_ATOMIC) { + if (!generic_atomic_write_valid(iocb->ki_pos, from, + i_blocksize(inode), + XFS_FSB_TO_B(mp, xfs_get_extsz(ip)))) { + return -EINVAL; + } + } + /* DIO must be aligned to device logical sector size */ if ((iocb->ki_pos | count) & target->bt_logical_sectormask) return -EINVAL;

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

--------------------------------

Support FS_IOC_SETATOMIC ioctl, if filesystem and file could support atomic write, then enable file's atomic write function. Database like Mysql could use this ioctl check and enable file atomic write.

Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/xfs_ioctl.c | 45 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/fs.h | 1 + 2 files changed, 46 insertions(+)

diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index e96ebea20991..fbaf3d0ddd8e 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -2112,6 +2112,26 @@ xfs_fs_eofblocks_from_user( return 0; }

+static int +xfs_ioc_set_atomic_write( + struct xfs_inode *ip) +{ + struct xfs_trans *tp; + int error; + + tp = xfs_ioctl_setattr_get_trans(ip, NULL); + if (IS_ERR(tp)) { + error = PTR_ERR(tp); + goto out; + } + + ip->i_d.di_flags2 |= XFS_DIFLAG2_ATOMICWRITES; + + error = xfs_trans_commit(tp); +out: + return error; +} + /* * Note: some of the ioctl's return positive numbers as a * byte count indicating success, such as readlink_by_handle. @@ -2139,6 +2159,31 @@ xfs_file_ioctl( return xfs_ioc_getlabel(mp, arg); case FS_IOC_SETFSLABEL: return xfs_ioc_setlabel(filp, mp, arg); + case FS_IOC_SETATOMIC: + if (!xfs_has_atomicwrites(mp)) + return -1; + if (!S_ISREG(inode->i_mode)) + return -1; + if (xfs_inode_atomicwrites(ip)) + return 0; + if (!xfs_inode_forcealign(ip)) + return -1; + + xfs_ilock(ip, XFS_IOLOCK_EXCL | XFS_MMAPLOCK_EXCL); + error = xfs_ioc_set_atomic_write(ip); + xfs_iunlock(ip, XFS_IOLOCK_EXCL | XFS_MMAPLOCK_EXCL); + if (error) { + xfs_alert(mp, "%s: set ino 0x%llx atomic write fail!", + __func__, XFS_I(inode)->i_ino); + return -1; + } else { + struct xfs_buftarg *target = xfs_inode_buftarg(ip); + + if ((filp->f_flags & O_DIRECT) && + bdev_can_atomic_write(target->bt_bdev)) + filp->f_mode |= FMODE_CAN_ATOMIC_WRITE; + return 0; + } case XFS_IOC_ALLOCSP: case XFS_IOC_FREESP: case XFS_IOC_ALLOCSP64: diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index ca8b6f6c479d..332b0709756b 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -217,6 +217,7 @@ struct fsxattr { #define FS_IOC_FSSETXATTR _IOW('X', 32, struct fsxattr) #define FS_IOC_GETFSLABEL _IOR(0x94, 49, char[FSLABEL_MAX]) #define FS_IOC_SETFSLABEL _IOW(0x94, 50, char[FSLABEL_MAX]) +#define FS_IOC_SETATOMIC _IOW(0x95, 2, uint)

/* * Inode flags (FS_IOC_GETFLAGS / FS_IOC_SETFLAGS)

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

--------------------------------

Since introduce "fs: xfs: Introduce FORCEALIGN inode flag", generic/079 report fail, the reason is that vfs_ioc_fssetxattr_check() would clear FS_XFLAG_EXTSIZE and FS_XFLAG_EXTSZINHERIT due to fa not update completely.

Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/xfs_ioctl.c | 4 ++++ 1 file changed, 4 insertions(+)

diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index fbaf3d0ddd8e..939b91124ce1 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -1649,6 +1649,10 @@ xfs_ioc_setxflags( }

xfs_fill_fsxattr(ip, false, &old_fa); + fa.fsx_extsize = old_fa.fsx_extsize; + fa.fsx_cowextsize = old_fa.fsx_cowextsize; + fa.fsx_projid = old_fa.fsx_projid; + fa.fsx_nextents = old_fa.fsx_nextents; error = vfs_ioc_fssetxattr_check(VFS_I(ip), &old_fa, &fa); if (error) { xfs_trans_cancel(tp);

From: John Garry john.g.garry@oracle.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/all/20240326133813.3224593-1-john.g.garry@oracle.com...

--------------------------------

Like we already do for rtvol, only free full extents for forcealign in xfs_free_file_space().

Signed-off-by: John Garry john.g.garry@oracle.com Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/xfs_bmap_util.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c index 94e5d57de432..5879f03b8660 100644 --- a/fs/xfs/xfs_bmap_util.c +++ b/fs/xfs/xfs_bmap_util.c @@ -928,8 +928,11 @@ xfs_free_file_space( startoffset_fsb = XFS_B_TO_FSB(mp, offset); endoffset_fsb = XFS_B_TO_FSBT(mp, offset + len);

- /* We can only free complete realtime extents. */ - if (XFS_IS_REALTIME_INODE(ip) && mp->m_sb.sb_rextsize > 1) { + /* Free only complete extents. */ + if (xfs_inode_forcealign(ip) && ip->i_d.di_extsize > 1) { + startoffset_fsb = roundup_64(startoffset_fsb, ip->i_d.di_extsize); + endoffset_fsb = rounddown_64(endoffset_fsb, ip->i_d.di_extsize); + } else if (XFS_IS_REALTIME_INODE(ip) && mp->m_sb.sb_rextsize > 1) { startoffset_fsb = roundup_64(startoffset_fsb, mp->m_sb.sb_rextsize); endoffset_fsb = rounddown_64(endoffset_fsb,

From: Zhang Yi yi.zhang@huawei.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/linux-fsdevel/20240529095206.2568162-1-yi.zhang@huaw...

--------------------------------

Add a new helper rem_u64() to only get the remainder of unsigned 64bit divide with 32bit divisor.

Signed-off-by: Zhang Yi yi.zhang@huawei.com Signed-off-by: Long Li leo.lilong@huawei.com --- include/linux/math64.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)

diff --git a/include/linux/math64.h b/include/linux/math64.h index 66deb1fdc2ef..b5c4d1df08e5 100644 --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -3,6 +3,7 @@ #define _LINUX_MATH64_H

#include <linux/types.h> +#include <linux/log2.h> #include <vdso/math64.h> #include <asm/div64.h>

@@ -11,6 +12,20 @@ #define div64_long(x, y) div64_s64((x), (y)) #define div64_ul(x, y) div64_u64((x), (y))

+/** + * rem_u64 - remainder of unsigned 64bit divide with 32bit divisor + * @dividend: unsigned 64bit dividend + * @divisor: unsigned 32bit divisor + * + * Return: dividend % divisor + */ +static inline u32 rem_u64(u64 dividend, u32 divisor) +{ + if (is_power_of_2(divisor)) + return dividend & (divisor - 1); + return dividend % divisor; +} + /** * div_u64_rem - unsigned 64bit divide with 32bit divisor with remainder * @dividend: unsigned 64bit dividend @@ -85,6 +100,15 @@ static inline s64 div64_s64(s64 dividend, s64 divisor) #define div64_long(x, y) div_s64((x), (y)) #define div64_ul(x, y) div_u64((x), (y))

+#ifndef rem_u64 +static inline u32 rem_u64(u64 dividend, u32 divisor) +{ + if (is_power_of_2(divisor)) + return dividend & (divisor - 1); + return do_div(dividend, divisor); +} +#endif + #ifndef div_u64_rem static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder) {

From: Zhang Yi yi.zhang@huawei.com

maillist inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

Reference: https://lore.kernel.org/linux-fsdevel/20240529095206.2568162-1-yi.zhang@huaw...

--------------------------------

When truncating down an inode, we call xfs_truncate_page() to zero out the tail partial block that beyond new EOF, which prevents exposing stale data. But xfs_truncate_page() always assumes the blocksize is i_blocksize(inode), it's not always true if we have a large allocation unit for a file and we should aligned to this unitsize, e.g. realtime inode should aligned to the rtextsize.

Current xfs_setattr_size() can't support zeroing out a large alignment size on trucate down since the process order is wrong. We first do zero out through xfs_truncate_page(), and then update inode size through truncate_setsize() immediately. If the zeroed range is larger than a folio, the write back path would not write back zeroed pagecache beyond the EOF folio, so it doesn't write zeroes to the entire tail extent and could expose stale data after an appending write into the next aligned extent.

We need to adjust the order to zero out tail aligned blocks, write back zeroed or cached data, update i_size and drop cache beyond aligned EOF block, preparing for the fix of realtime inode and supporting the upcoming forced alignment feature.

Signed-off-by: Zhang Yi yi.zhang@huawei.com Signed-off-by: Long Li leo.lilong@huawei.com --- fs/xfs/xfs_iops.c | 115 ++++++++++++++++++++++++---------------------- 1 file changed, 59 insertions(+), 56 deletions(-)

diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index 092fb02d1a13..b13d3c458c46 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -769,7 +769,7 @@ xfs_setattr_size( int error; uint lock_flags = 0; bool did_zeroing = false; - unsigned int blocksize = i_blocksize(inode); + bool write_back = false;

ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL)); ASSERT(xfs_isilocked(ip, XFS_MMAPLOCK_EXCL)); @@ -806,21 +806,10 @@ xfs_setattr_size( */ inode_dio_wait(inode);

- /* - * File data changes must be complete before we start the transaction to - * modify the inode. This needs to be done before joining the inode to - * the transaction because the inode cannot be unlocked once it is a - * part of the transaction. - * - * Start with zeroing any data beyond EOF that we may expose on file - * extension, or zeroing out the rest of the block on a downward - * truncate. - */ - if (newsize > oldsize) { - trace_xfs_zero_eof(ip, oldsize, newsize - oldsize); - error = iomap_zero_range(inode, oldsize, newsize - oldsize, - &did_zeroing, &xfs_buffered_write_iomap_ops); - } else { + write_back = newsize > ip->i_d.di_size && oldsize != ip->i_d.di_size; + if (newsize < oldsize) { + unsigned int blocksize = i_blocksize(inode); + /* * iomap won't detect a dirty page over an unwritten block (or a * cow block over a hole) and subsequently skips zeroing the @@ -828,53 +817,67 @@ xfs_setattr_size( * convert the block before the pagecache truncate. */ error = filemap_write_and_wait_range(inode->i_mapping, newsize, - newsize); + roundup_64(newsize, blocksize) - 1); if (error) return error; error = iomap_truncate_page(inode, newsize, blocksize, &did_zeroing, &xfs_buffered_write_iomap_ops); - }

- if (error) - return error; + /* + * We are going to log the inode size change in this transaction + * so any previous writes that are beyond the on disk EOF and + * the new EOF that have not been written out need to be written + * here. If we do not write the data out, we expose ourselves + * to the null files problem. Note that this includes any block + * zeroing we did above; otherwise those blocks may not be + * zeroed after a crash. + */ + if (did_zeroing || write_back) { + error = filemap_write_and_wait_range(inode->i_mapping, + min_t(loff_t, ip->i_d.di_size, newsize), + roundup_64(newsize, blocksize) - 1); + if (error) + return error; + }

- /* - * We've already locked out new page faults, so now we can safely remove - * pages from the page cache knowing they won't get refaulted until we - * drop the XFS_MMAP_EXCL lock after the extent manipulations are - * complete. The truncate_setsize() call also cleans partial EOF page - * PTEs on extending truncates and hence ensures sub-page block size - * filesystems are correctly handled, too. - * - * We have to do all the page cache truncate work outside the - * transaction context as the "lock" order is page lock->log space - * reservation as defined by extent allocation in the writeback path. - * Hence a truncate can fail with ENOMEM from xfs_trans_alloc(), but - * having already truncated the in-memory version of the file (i.e. made - * user visible changes). There's not much we can do about this, except - * to hope that the caller sees ENOMEM and retries the truncate - * operation. - * - * And we update in-core i_size and truncate page cache beyond newsize - * before writeback the [di_size, newsize] range, so we're guaranteed - * not to write stale data past the new EOF on truncate down. - */ - truncate_setsize(inode, newsize); + /* + * Updating i_size after writing back to make sure the zeroed + * blocks could been written out, and drop all the page cache + * range that beyond blocksize aligned new EOF block. + * + * We've already locked out new page faults, so now we can + * safely remove pages from the page cache knowing they won't + * get refaulted until we drop the XFS_MMAP_EXCL lock after the + * extent manipulations are complete. + */ + i_size_write(inode, newsize); + truncate_pagecache(inode, roundup_64(newsize, blocksize)); + } else { + /* + * Start with zeroing any data beyond EOF that we may expose on + * file extension. + */ + if (newsize > oldsize) { + trace_xfs_zero_eof(ip, oldsize, newsize - oldsize); + error = iomap_zero_range(inode, oldsize, newsize - oldsize, + &did_zeroing, &xfs_buffered_write_iomap_ops); + if (error) + return error; + }

- /* - * We are going to log the inode size change in this transaction so - * any previous writes that are beyond the on disk EOF and the new - * EOF that have not been written out need to be written here. If we - * do not write the data out, we expose ourselves to the null files - * problem. Note that this includes any block zeroing we did above; - * otherwise those blocks may not be zeroed after a crash. - */ - if (did_zeroing || - (newsize > ip->i_d.di_size && oldsize != ip->i_d.di_size)) { - error = filemap_write_and_wait_range(VFS_I(ip)->i_mapping, - ip->i_d.di_size, newsize - 1); - if (error) - return error; + /* + * The truncate_setsize() call also cleans partial EOF page + * PTEs on extending truncates and hence ensures sub-page block + * size filesystems are correctly handled, too. + */ + truncate_setsize(inode, newsize); + + if (did_zeroing || write_back) { + error = filemap_write_and_wait_range(inode->i_mapping, + ip->i_d.di_size, newsize - 1); + if (error) + return error; + } }

error = xfs_trans_alloc(mp, &M_RES(mp)->tr_itruncate, 0, 0, 0, &tp);

hulk inclusion category: feature bugzilla: https://gitee.com/openeuler/kernel/issues/I9VTE3 CVE: NA

--------------------------------

Fix kabi breakage in struct queue_limits.

Signed-off-by: Long Li leo.lilong@huawei.com --- block/blk-core.c | 16 ++++++++++++++- block/blk-merge.c | 2 +- block/blk-settings.c | 37 +++++++++++++++++++++++++---------- block/blk-sysfs.c | 1 + block/blk.h | 1 + drivers/md/dm-table.c | 1 + drivers/nvme/host/core.c | 12 ++++++++---- include/linux/blkdev.h | 42 +++++++++++++++++++++++----------------- 8 files changed, 78 insertions(+), 34 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c index 27f356c07d26..df6564d773e6 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -81,6 +81,7 @@ __setup("precise_iostat=", precise_iostat_setup); * For queue allocation */ struct kmem_cache *blk_requestq_cachep; +struct kmem_cache *queue_atomic_write_cachep;

/* * Controlling structure to kblockd @@ -532,6 +533,7 @@ static void blk_timeout_work(struct work_struct *work) struct request_queue *blk_alloc_queue(int node_id) { struct request_queue *q; + struct queue_atomic_write_limits *aw_limits; int ret;

q = kmem_cache_alloc_node(blk_requestq_cachep, @@ -539,11 +541,18 @@ struct request_queue *blk_alloc_queue(int node_id) if (!q) return NULL;

+ aw_limits = kmem_cache_alloc_node(queue_atomic_write_cachep, + GFP_KERNEL | __GFP_ZERO, node_id); + if (!aw_limits) + goto fail_q; + + q->limits.aw_limits = aw_limits; + q->last_merge = NULL;

q->id = ida_simple_get(&blk_queue_ida, 0, 0, GFP_KERNEL); if (q->id < 0) - goto fail_q; + goto fail_aw;

ret = bioset_init(&q->bio_split, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS); if (ret) @@ -594,6 +603,7 @@ struct request_queue *blk_alloc_queue(int node_id)

blk_queue_dma_alignment(q, 511); blk_set_default_limits(&q->limits); + blk_set_default_atomic_write_limits(&q->limits); q->nr_requests = BLKDEV_MAX_RQ;

return q; @@ -608,6 +618,8 @@ struct request_queue *blk_alloc_queue(int node_id) bioset_exit(&q->bio_split); fail_id: ida_simple_remove(&blk_queue_ida, q->id); +fail_aw: + kmem_cache_free(queue_atomic_write_cachep, aw_limits); fail_q: kmem_cache_free(blk_requestq_cachep, q); return NULL; @@ -1923,6 +1935,8 @@ int __init blk_dev_init(void)

blk_requestq_cachep = kmem_cache_create("request_queue", sizeof(struct request_queue), 0, SLAB_PANIC, NULL); + queue_atomic_write_cachep = kmem_cache_create("queue_atomic_write", + sizeof(struct queue_atomic_write_limits), 0, SLAB_PANIC, NULL);

blk_debugfs_root = debugfs_create_dir("block", NULL);

diff --git a/block/blk-merge.c b/block/blk-merge.c index 7ca680a6c037..3b2004308e93 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -195,7 +195,7 @@ static inline unsigned get_max_io_size(struct request_queue *q, * it may less than the bio size, which we cannot tolerate. */ if (bio->bi_opf & REQ_ATOMIC) - max_sectors = q->limits.atomic_write_max_sectors; + max_sectors = q->limits.aw_limits->atomic_write_max_sectors; else max_sectors = sectors;

diff --git a/block/blk-settings.c b/block/blk-settings.c index 754b758946a9..8d378d62178c 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -63,6 +63,20 @@ void blk_set_default_limits(struct queue_limits *lim) } EXPORT_SYMBOL(blk_set_default_limits);

+void blk_set_default_atomic_write_limits(struct queue_limits *lim) +{ + if (lim->aw_limits) { + lim->aw_limits->atomic_write_hw_max = 0; + lim->aw_limits->atomic_write_max_sectors = 0; + lim->aw_limits->atomic_write_hw_boundary = 0; + lim->aw_limits->atomic_write_hw_unit_min = 0; + lim->aw_limits->atomic_write_unit_min = 0; + lim->aw_limits->atomic_write_hw_unit_max = 0; + lim->aw_limits->atomic_write_unit_max = 0; + } +} +EXPORT_SYMBOL(blk_set_default_atomic_write_limits); + /** * blk_set_stacking_limits - set default limits for stacking devices * @lim: the queue_limits structure to reset @@ -153,13 +167,16 @@ static void blk_atomic_writes_update_limits(struct queue_limits *limits)

unit_limit = rounddown_pow_of_two(unit_limit);

- limits->atomic_write_max_sectors = - min(limits->atomic_write_hw_max >> SECTOR_SHIFT, + if (!limits->aw_limits) + return; + + limits->aw_limits->atomic_write_max_sectors = + min(limits->aw_limits->atomic_write_hw_max >> SECTOR_SHIFT, limits->max_hw_sectors); - limits->atomic_write_unit_min = - min(limits->atomic_write_hw_unit_min, unit_limit); - limits->atomic_write_unit_max = - min(limits->atomic_write_hw_unit_max, unit_limit); + limits->aw_limits->atomic_write_unit_min = + min(limits->aw_limits->atomic_write_hw_unit_min, unit_limit); + limits->aw_limits->atomic_write_unit_max = + min(limits->aw_limits->atomic_write_hw_unit_max, unit_limit); }

/** @@ -243,7 +260,7 @@ EXPORT_SYMBOL(blk_queue_max_discard_sectors); void blk_queue_atomic_write_max_bytes(struct request_queue *q, unsigned int bytes) { - q->limits.atomic_write_hw_max = bytes; + q->limits.aw_limits->atomic_write_hw_max = bytes; blk_atomic_writes_update_limits(&q->limits); } EXPORT_SYMBOL(blk_queue_atomic_write_max_bytes); @@ -257,7 +274,7 @@ EXPORT_SYMBOL(blk_queue_atomic_write_max_bytes); void blk_queue_atomic_write_boundary_bytes(struct request_queue *q, unsigned int bytes) { - q->limits.atomic_write_hw_boundary = bytes; + q->limits.aw_limits->atomic_write_hw_boundary = bytes; } EXPORT_SYMBOL(blk_queue_atomic_write_boundary_bytes);

@@ -270,7 +287,7 @@ EXPORT_SYMBOL(blk_queue_atomic_write_boundary_bytes); void blk_queue_atomic_write_unit_min_bytes(struct request_queue *q, unsigned int bytes) { - q->limits.atomic_write_hw_unit_min = bytes; + q->limits.aw_limits->atomic_write_hw_unit_min = bytes; blk_atomic_writes_update_limits(&q->limits); } EXPORT_SYMBOL(blk_queue_atomic_write_unit_min_bytes); @@ -284,7 +301,7 @@ EXPORT_SYMBOL(blk_queue_atomic_write_unit_min_bytes); void blk_queue_atomic_write_unit_max_bytes(struct request_queue *q, unsigned int bytes) { - q->limits.atomic_write_hw_unit_max = bytes; + q->limits.aw_limits->atomic_write_hw_unit_max = bytes; blk_atomic_writes_update_limits(&q->limits); } EXPORT_SYMBOL(blk_queue_atomic_write_unit_max_bytes); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 6916f1ed4590..8e35d63b30c2 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -764,6 +764,7 @@ static void blk_free_queue_rcu(struct rcu_head *rcu_head) rcu_head);

percpu_ref_exit(&q->q_usage_counter); + kmem_cache_free(queue_atomic_write_cachep, q->limits.aw_limits); kmem_cache_free(blk_requestq_cachep, q); }

diff --git a/block/blk.h b/block/blk.h index 4bbcc971d4f7..1bdd07040d42 100644 --- a/block/blk.h +++ b/block/blk.h @@ -29,6 +29,7 @@ struct blk_flush_queue { };

extern struct kmem_cache *blk_requestq_cachep; +extern struct kmem_cache *queue_atomic_write_cachep; extern struct kobj_type blk_queue_ktype; extern struct ida blk_queue_ida;

diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index eb95b5ce7b8f..d8f54d2a74d9 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1816,6 +1816,7 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, /* * Copy table's limits to the DM device's request_queue */ + limits->aw_limits = q->limits.aw_limits; q->limits = *limits;

if (dm_table_supports_nowait(t)) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 4fa0c55787d2..76626f5a737f 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -2052,10 +2052,14 @@ static void nvme_update_atomic_write_disk_info(struct nvme_ns *ns, if (le16_to_cpu(id->nabspf)) boundary = (le16_to_cpu(id->nabspf) + 1) * bs; } - lim->atomic_write_hw_max = atomic_bs; - lim->atomic_write_hw_boundary = boundary; - lim->atomic_write_hw_unit_min = bs; - lim->atomic_write_hw_unit_max = rounddown_pow_of_two(atomic_bs); + + if (!lim->aw_limits) + return; + + lim->aw_limits->atomic_write_hw_max = atomic_bs; + lim->aw_limits->atomic_write_hw_boundary = boundary; + lim->aw_limits->atomic_write_hw_unit_min = bs; + lim->aw_limits->atomic_write_hw_unit_max = rounddown_pow_of_two(atomic_bs); }

static void nvme_set_queue_limits(struct nvme_ctrl *ctrl, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index a53e70a6aeec..31e8e98d4990 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -323,6 +323,17 @@ enum blk_zoned_model { BLK_ZONED_HM, /* Host-managed zoned block device */ };

+struct queue_atomic_write_limits { + /* atomic write limits */ + unsigned int atomic_write_hw_max; + unsigned int atomic_write_max_sectors; + unsigned int atomic_write_hw_boundary; + unsigned int atomic_write_hw_unit_min; + unsigned int atomic_write_unit_min; + unsigned int atomic_write_hw_unit_max; + unsigned int atomic_write_unit_max; +}; + struct queue_limits { unsigned long bounce_pfn; unsigned long seg_boundary_mask; @@ -346,16 +357,6 @@ struct queue_limits { unsigned int discard_granularity; unsigned int discard_alignment;

- /* atomic write limits */ - unsigned int atomic_write_hw_max; - unsigned int atomic_write_max_sectors; - unsigned int atomic_write_hw_boundary; - unsigned int atomic_write_hw_unit_min; - unsigned int atomic_write_unit_min; - unsigned int atomic_write_hw_unit_max; - unsigned int atomic_write_unit_max; - - unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -365,7 +366,11 @@ struct queue_limits { unsigned char raid_partial_stripes_expensive; enum blk_zoned_model zoned;

+#ifndef __GENKSYMS__ + struct queue_atomic_write_limits *aw_limits; +#else KABI_RESERVE(1) +#endif };

typedef int (*report_zones_cb)(struct blk_zone *zone, unsigned int idx, @@ -1108,7 +1113,7 @@ static inline unsigned int blk_queue_get_max_sectors(struct request *rq) return q->limits.max_write_zeroes_sectors;

if (rq->cmd_flags & REQ_ATOMIC) - return q->limits.atomic_write_max_sectors; + return q->limits.aw_limits->atomic_write_max_sectors;

return q->limits.max_sectors; } @@ -1215,6 +1220,7 @@ extern void blk_limits_io_opt(struct queue_limits *limits, unsigned int opt); extern void blk_queue_io_opt(struct request_queue *q, unsigned int opt); extern void blk_set_queue_depth(struct request_queue *q, unsigned int depth); extern void blk_set_default_limits(struct queue_limits *lim); +extern void blk_set_default_atomic_write_limits(struct queue_limits *lim); extern void blk_set_stacking_limits(struct queue_limits *lim); extern int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, sector_t offset); @@ -1673,25 +1679,25 @@ static inline unsigned int bdev_max_active_zones(struct block_device *bdev) static inline unsigned int queue_atomic_write_unit_max_bytes(const struct request_queue *q) { - return q->limits.atomic_write_unit_max; + return q->limits.aw_limits->atomic_write_unit_max; }

static inline unsigned int queue_atomic_write_unit_min_bytes(const struct request_queue *q) { - return q->limits.atomic_write_unit_min; + return q->limits.aw_limits->atomic_write_unit_min; }

static inline unsigned int queue_atomic_write_boundary_bytes(const struct request_queue *q) { - return q->limits.atomic_write_hw_boundary; + return q->limits.aw_limits->atomic_write_hw_boundary; }

static inline unsigned int queue_atomic_write_max_bytes(const struct request_queue *q) { - return q->limits.atomic_write_max_sectors << SECTOR_SHIFT; + return q->limits.aw_limits->atomic_write_max_sectors << SECTOR_SHIFT; }

static inline int queue_dma_alignment(const struct request_queue *q) @@ -2152,14 +2158,14 @@ static inline bool bdev_can_atomic_write(struct block_device *bdev) struct request_queue *bd_queue = bdev_get_queue(bdev); struct queue_limits *limits = &bd_queue->limits;

- if (!limits->atomic_write_unit_min) + if (!limits->aw_limits->atomic_write_unit_min) return false;

if (bdev_is_partition(bdev)) { sector_t bd_start_sect = bdev->bd_part->start_sect; unsigned int alignment = - max(limits->atomic_write_unit_min, - limits->atomic_write_hw_boundary); + max(limits->aw_limits->atomic_write_unit_min, + limits->aw_limits->atomic_write_hw_boundary); if (!IS_ALIGNED(bd_start_sect, alignment)) return false; }