NVMe初心者のメモです。

環境

OS: CentOS7
NVMe Intel P3600
Kernel: 4.4 (自身でリビルド)

参考資料

How to use and benchmark NVMe SSD, create a Software RAID and analyze performance – The answers are here
のPDFが参考になる。

特に参考にしたPDF

Hands-on Lab: How to Unleash Your Storage Performance by Using NVM ExpressTM Based PCI Express® Solid-State Drives

文中に記載されているリンク

PCIe SSD PACC EP P3700 Intel® Solid-State Drive Data Center Tool 富士通の日本語ドキュメント

インテル® Solid-State Drive データセンター・ツールのユーザーおよびインストールガイド

データセンターツールの導入

インテル® Solid-State Drive データセンター・ツールからファイルをダウンロード

unzip ../DataCenterTool_3_0_0_Linux.zip 
Archive:  ../DataCenterTool_3_0_0_Linux.zip
  inflating: isdct-3.0.0.400-15.i386.rpm  
  inflating: isdct-3.0.0.400-15.x86_64.rpm  
  inflating: NSGSoftwareLicenseAgreement__EndUserNoModify__2016.txt

isdct-3.0.0.400-15.x86_64.rpmをインストール

rpm -ivh isdct-3.0.0.400-15.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:isdct-3.0.0.400-15               ################################# [100%]

rpm -ql isdct
/etc/isdct
/usr/bin/isdct
/usr/lib/isdct/FirmwareModules/generic_firmware_module.so
/usr/lib/isdct/libIntel.SSDFeatures.so.2.0.0
/usr/lib/isdct/libLSI.AdapterAPI.so.4.02

Help:
    help [-help|-h] [-output|-o (text|nvmxml|json)] [Name = (name)] [verb =
    (verb)]
IntelSSD:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -intelssd [(Index|SerialNumber)]
EnduranceAnalyzer:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) EnduranceAnalyzer = ('reset')
PhyConfig:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PhyConfig = (0|1|2|3)
PhysicalSectorSize:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PhysicalSectorSize = (512|4096)
PLITestTimeInterval:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PLITestTimeInterval = (0-6)
PowerGovernorMode:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PowerGovernorMode = (0|1|2)
ReadErrorRecoveryTime:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) ReadErrorRecoveryTimer = (value)
PhySpeed:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) PhySpeed = (1.5|3|6)
SSCEnabled:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) SSCEnabled = ('true'|'false')
TemperatureLoggingInterval:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) TemperatureLoggingInterval = (time)
TempThreshold:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) TempThreshold = (value)
WriteErrorRecoveryTime:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteErrorRecoveryTimer = (value)
WriteCacheState:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteCacheState = (1|2|3)
MaximumLBA:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) MaximumLBA = (numGB|1-100%|LBA|'native')
FirmwareUpdate:
    load [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber)
SMART:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -smart [(id)] [-intelssd [(Index|SerialNumber)]]
Sensors:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -sensor [-intelssd [(Index|SerialNumber)]]
Performance:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -performance [-intelssd [(Index|SerialNumber)]]
NVMeLog:
    show [-help|-h] [-output|-o (text|nvmxml|json)] [-intelssd
    [(Index|SerialNumber)]] -nvmelog
    [('ErrorInfo'|'SmartHealthInfo'|'FirmwareSlotInfo'|'TemperatureStatistics')]
Delete:
    delete [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber)
SMBusAddress:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) SMBusAddress = (address)
LatencyTrackingEnabled:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) LatencyTrackingEnabled = (true|false)
DIPM:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) DIPMEnabled = ('true' | 'false')
WriteCache:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteCacheEnabled = ('true' | 'false')
WriteCacheReorderingState:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) WriteCacheReorderingStateEnabled = ('true'|'false')
CreateNamespace:
    create [-help|-h] [-output|-o (text|nvmxml|json)] -namespace -intelssd
    (Index|SerialNumber) Size = (blocks) [LBAFormat = (0-NumLBAFormats)]
    [ProtectionInformation = (0|1)] [MultiPathIoCapabilities = (0|1)]
DeleteNamespace:
    delete [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -namespace
    (namespace id) -intelssd (Index|SerialNumber)
AttachNamespace:
    attach [-help|-h] [-output|-o (text|nvmxml|json)] -namespace (namespace id)
    -intelssd (Index|SerialNumber) [-nvmecontroller (controller ID integer)]
DetachNamespace:
    detach [-help|-h] [-output|-o (text|nvmxml|json)] -namespace (namespace id)
    -intelssd (Index|SerialNumber) [-nvmecontroller (controller ID integer)]
IdentifyDevice:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -identify [-namespace
    [(integer | 'attached' | 'allocated')]] [-nvmecontroller] [-intelssd
    [(Index|SerialNumber)]]
NVMeFormat:
    start [-help|-h] [-force|-f] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) -nvmeformat [-namespace (namespace id)] [LBAFormat =
    (0-NumLBAFormats)] [SecureEraseSetting = (0|1|2)] [ProtectionInformation =
    (0|1)] [MetadataSettings = (0|1)]
LatencyStatistics:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -latencystatistics
    ('reads'|'writes') [-intelssd [(Index|SerialNumber)]]
HDATemperature:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -hdatemperature [-intelssd
    [(Index|SerialNumber)]]
Nlog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -nlog
AssertLog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -assertlog
EventLog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -eventlog
SATAGeneralPurposeLog:
    dump [-help|-h] [-destination (path)] [-output|-o (text|nvmxml|json)]
    [-intelssd [(Index|SerialNumber)]] -gpl (log address) [PageNum = [(page
    #)]] [SectorCount = [(sectors)]]
PhyCounters:
    show [-help|-h] [-output|-o (text|nvmxml|json)] -phycounters [-intelssd
    [(Index|SerialNumber)]]
DriveSelfTest:
    start [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) -selftest [('short'|'extended'|'conveyance')]
Standby:
    start [-help|-h] [-output|-o (text|nvmxml|json)] -intelssd
    (Index|SerialNumber) -standby
NVMeControllerList:
    show [-help|-h] [-output|-o (text|nvmxml|json)] [-namespace (namespace id)]
    -nvmecontroller [-intelssd [(Index|SerialNumber)]]
EnableLog:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -system EnableLog =
    ('true'|'false')
EnableLSIAdapter:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -system EnableLSIAdapter =
    ('true'|'false')
LogFile:
    set [-help|-h] [-output|-o (text|nvmxml|json)] -system LogFile = (file name)
System:
    show [-help|-h] [-display|-d (Property1,...)] [-all|-a] [-output|-o
    (text|nvmxml|json)] -system
Version:
    version [-all|-a] [-display|-d (Property1,...)] [-help|-h] [-output|-o
    (text|nvmxml|json)]

isdct show -intelssd

- Intel SSD DC P3600 Series CVMD5395004X400AGN -

Bootloader : 8B1B012E
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
Firmware : 8DV10131
FirmwareUpdateAvailable : Firmware=8DV10171 Bootloader=8B1B0131
Index : 0
ModelNumber : INTEL SSDPEDME400G4
ProductFamily : Intel SSD DC P3600 Series
SerialNumber : XXXXXXXXXXX

- Intel SSD DC P3600 Series CVMD5395007L400AGN -

Bootloader : 8B1B012E
DevicePath : /dev/nvme1n1
DeviceStatus : Healthy
Firmware : 8DV10131
FirmwareUpdateAvailable : Firmware=8DV10171 Bootloader=8B1B0131
Index : 1
ModelNumber : INTEL SSDPEDME400G4
ProductFamily : Intel SSD DC P3600 Series
SerialNumber : XXXXXXXXXXX

なるほど

RAID0

mdadm -C /dev/md/imsm0 /dev/nvme0n1 /dev/nvme1n1 -n 2 -e imsm -f 
mdadm: /dev/nvme0n1 appears to be part of a raid array:
       level=raid0 devices=0 ctime=Thu Jan  1 09:00:00 1970
Continue creating array? y
mdadm: container /dev/md/imsm0 prepared.

mdadm -C /dev/md0 /dev/md/imsm0 -n 2 -l 0 -c 128 -f 
mdadm: array /dev/md0 started.

parted /dev/md0 mklabel gpt
Warning: The existing disk label on /dev/md0 will be destroyed and all data on
this disk will be lost. Do you want to continue?
Yes/No? yes                                                               
Information: You may need to update /etc/fstab.

parted /dev/md0                                        

(parted) mkpart                                                           
Partition name?  []?                                                      
File system type?  [ext2]? xfs                                            
Start? 0%                                                                 
End? -1

(parted) p                                                                
Model: Linux Software RAID Array (md)
Disk /dev/md0: 800GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  800GB  800GB

(parted) quit                                                             
Information: You may need to update /etc/fstab.

/sbin/mkfs.xfs -K /dev/md0p1 -f

mount -o noatime,nodiratime,nobarrier /dev/md0p1 /mnt/nvme1

mdadm --detail --scan > /etc/mdadm.conf

mdadm.confの例

ARRAY /dev/md/imsm0 metadata=imsm UUID=a66e2dd4:e3167963:92df1c07:69c7f299
ARRAY /dev/md0 container=/dev/md/imsm0 member=0 UUID=0c3f3fbd:a1d6986b:b18d262b:f3ae58ff

/proc/mdstatの値

cat /proc/mdstat 
Personalities : [raid0] 
md127 : inactive nvme0n1[1](S) nvme1n1[0](S)
      6306 blocks super external:imsm
       
md0 : active raid0 nvme1n1[1] nvme0n1[0]
      781416448 blocks super external:/md127/0 128k chunks
      
unused devices: <none>

RAID0の手動停止

mdadm --stop /dev/md0
mdadm: stopped /dev/md0

mdadm --stop /dev/md/imsm0 
mdadm: stopped /dev/md/imsm0

cat /proc/mdstat 
Personalities : [raid0] 
unused devices: <none>

RAID0の手動開始

mdadm --assemble --scan
mdadm: Container /dev/md/imsm0 has been assembled with 2 drives
mdadm: Started /dev/md0 with 2 devices

madm.confを書き忘れた場合の手動アッセンブル

mdadm --assemble /dev/md/imsm0 /dev/nvme0n1 /dev/nvme1n1

mdadm --assemble /dev/md0 /dev/md/imsm0