Hi all,
We are migrating a SCSI HBA driver from KEXT to DriverKit (DEXT), with our DEXT inheriting from IOUserSCSIParallelInterfaceController. We've encountered a data corruption issue that is reliably reproducible under specific conditions and are hoping for some assistance from the community.
Hardware and Driver Configuration:
Controller: LSI 3108
DEXT Configuration: We are reporting our hardware limitations to the framework via the UserReportHBAConstraints function, with the following key settings:
// UserReportHBAConstraints...
addConstraint(kIOMaximumSegmentAddressableBitCountKey, 0x20); // 32-bit
addConstraint(kIOMaximumSegmentCountWriteKey, 129);
addConstraint(kIOMaximumByteCountWriteKey, 0x80000); // 512KB
Observed Behavior: Direct I/O vs. Buffered I/O
We've observed that the I/O behavior differs drastically depending on whether it goes through the system file cache:
1. Direct I/O (Bypassing System Cache) -> 100% Successful
When we use fio with the direct=1 flag, our read/write and data verification tests pass perfectly for all file sizes, including 20GB+.
2. Buffered I/O (Using System Cache) -> 100% Failure at >128MB
Whether we use the standard cp command or fio with the direct=1 option removed to simulate buffered I/O, we observe the exact same, clear failure threshold:
Test Results:
File sizes ≤ 128MB: Success. Data checksums match perfectly.
File sizes ≥ 256MB: Failure. Checksums do not match, and the destination file is corrupted.
Evidence of failure reproduced with fio (buffered_integrity_test.fio, with direct=1 removed):
fio --size=128M buffered_integrity_test.fio -> Test Succeeded (err=0).
fio --size=256M buffered_integrity_test.fio -> Test Failed (err=92), reporting the following error, which proves a data mismatch during the verification phase:
verify: bad header ... at file ... offset 1048576, length 1048576
fio: ... error=Illegal byte sequence
Our Analysis and Hypothesis
The phenomenon of "Direct I/O succeeding while Buffered I/O fails" suggests the problem may be related to the cache synchronization mechanism at the end of the I/O process:
Our UserProcessParallelTask_Impl function correctly handles READ and WRITE commands.
When cp or fio (buffered) runs, the WRITE commands are successfully written to the LSI 3108 controller's onboard DRAM cache, and success is reported up the stack.
At the end of the operation, to ensure data is flushed to disk, the macOS file system issues an fsync, which is ultimately translated into a SYNCHRONIZE CACHE SCSI command (Opcode 0x35 or 0x91) and sent to our UserProcessParallelTask_Impl.
We hypothesize that our code may not be correctly identifying or handling this SYNCHRONIZE CACHE opcode. It might be reporting "success" up the stack without actually commanding the hardware to flush its cache to the physical disk.
The OS receives this "success" status and assumes the operation is safely complete.
In reality, however, the last batch of data remains only in the controller's volatile DRAM cache and is eventually lost.
This results in an incomplete or incorrect file tail, and while the file size may be correct, the data checksum will inevitably fail.
Summary
Our DEXT driver performs correctly when handling Direct I/O but consistently fails with data corruption when handling Buffered I/O for files larger than 128MB. We can reliably reproduce this issue using fio with the direct=1 option removed.
The root cause is very likely the improper handling of the SYNCHRONIZE CACHE command within our UserProcessParallelTask. P.S. This issue did not exist in the original KEXT version of the driver.
We would appreciate any advice or guidance on this issue.
Thank you.
