ある日突然mysqlが死んでしまった。以下そのときの対応手順メモ。
再起動しようとしても表題のようなエラーをログに吐いて起動できない。
エラーログの内容な以下のようなものだった。
140110 12:44:00 [Note] Plugin 'FEDERATED' is disabled.
140110 12:44:03 InnoDB: The InnoDB memory heap is disabled
140110 12:44:03 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140110 12:44:03 InnoDB: Compressed tables use zlib 1.2.3.4
140110 12:44:03 InnoDB: Initializing buffer pool, size = 256.0M
140110 12:44:03 InnoDB: Completed initialization of buffer pool
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 5.
InnoDB: You may have to recover from a backup.
140110 12:44:03 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 1ab7deac000000050000000000000000000000006820f2d60007000000000000000000000000000000000 ...
InnoDB: End of page dump
140110 12:44:03 InnoDB: Page checksum 448257708, prior-to-4.0.14-form checksum 3744172499
InnoDB: stored checksum 448257708, prior-to-4.0.14-form stored checksum 3926214354
InnoDB: Page lsn 0 1746989782, low 4 bytes of lsn at page end 1746989772
InnoDB: Page number (if stored to page already) 5,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a transaction system page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 5.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
140110 12:44:03 InnoDB: Assertion failure in thread 140677102274368 in file buf0buf.c line 3631
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
03:44:03 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346685 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7ff1f10c2349]
/usr/sbin/mysqld(handle_fatal_signal+0x483)[0x7ff1f0f87f33]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7ff1efcd4cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7ff1ef340425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7ff1ef343b8b]
/usr/sbin/mysqld(+0x645e11)[0x7ff1f11b1e11]
/usr/sbin/mysqld(+0x651688)[0x7ff1f11bd688]
/usr/sbin/mysqld(+0x65202c)[0x7ff1f11be02c]
/usr/sbin/mysqld(+0x6418b6)[0x7ff1f11ad8b6]
/usr/sbin/mysqld(+0x613e3c)[0x7ff1f117fe3c]
/usr/sbin/mysqld(+0x61605c)[0x7ff1f118205c]
/usr/sbin/mysqld(+0x601e2a)[0x7ff1f116de2a]
/usr/sbin/mysqld(+0x5cba29)[0x7ff1f1137a29]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x41)[0x7ff1f0f8a601]
/usr/sbin/mysqld(+0x30b581)[0x7ff1f0e77581]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xa94)[0x7ff1f0e7aba4]
/usr/sbin/mysqld(+0x284796)[0x7ff1f0df0796]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x59b)[0x7ff1f0df3f3b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7ff1ef32b76d]
/usr/sbin/mysqld(+0x27e2c5)[0x7ff1f0dea2c5]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
ファイルシステムのエラーがあったらしく、再起動したらdmesgに以下のようなログが吐かれていた。
[ 1.713380] EXT4-fs (vda1): INFO: recovery required on readonly filesystem
[ 1.719270] EXT4-fs (vda1): write access will be enabled during recovery
[ 1.728212] Refined TSC clocksource calibration: 2400.060 MHz.
[ 5.222477] input: QEMU 0.12.1 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input2
[ 5.233164] generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 0.12.1 QEMU USB Tablet] on usb-0000:00:01.2-1/input0
[ 5.243370] usbcore: registered new interface driver usbhid
[ 5.258301] usbhid: USB HID core driver
[ 5.343124] EXT4-fs (vda1): orphan cleanup on readonly fs
[ 5.348068] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131140
[ 5.348159] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131135
[ 5.348174] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131126
[ 5.348193] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131112
[ 5.348211] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131132
[ 5.348227] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131123
[ 5.348246] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131103
[ 5.348265] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131234
[ 5.348297] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131229
[ 5.348317] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131094
[ 5.348521] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131146
[ 5.348554] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131147
[ 5.348587] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131127
[ 5.348604] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131097
[ 5.348633] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131241
[ 5.348711] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131113
[ 5.348745] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131141
[ 5.348772] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131235
[ 5.348800] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131106
[ 5.348826] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131231
[ 5.348856] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131088
[ 5.348887] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131239
[ 5.348916] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131236
[ 5.348940] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131224
[ 5.348957] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131196
[ 5.348974] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131228
[ 5.348989] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131227
[ 5.349004] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131232
[ 5.349019] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131117
[ 5.349035] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131136
[ 5.349050] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131222
[ 5.349065] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131129
[ 5.349081] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131098
[ 5.349135] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131128
[ 5.349154] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131125
[ 5.349170] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131188
[ 5.349186] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131120
[ 5.349202] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131133
[ 5.353225] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131207
[ 5.353294] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131164
[ 5.353384] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131118
[ 5.353441] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131230
[ 5.353472] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131210
[ 5.353492] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131092
[ 5.353509] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131130
[ 5.353524] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131090
[ 5.353549] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131223
[ 5.353578] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131122
[ 5.353609] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131220
[ 5.353639] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131181
[ 5.353669] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131203
[ 5.353699] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131134
[ 5.353727] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131226
[ 5.353744] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131225
[ 5.353772] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131219
[ 5.353788] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131221
[ 5.353831] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131217
[ 5.353848] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131158
[ 5.353864] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131116
[ 5.353882] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131191
[ 5.353914] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131208
[ 5.353943] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131202
[ 5.353965] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131199
[ 5.353982] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131137
[ 5.353998] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131216
[ 5.354026] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131213
[ 5.354060] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131121
[ 5.354084] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131193
[ 5.354099] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131194
[ 5.354115] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131212
[ 5.354131] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131211
[ 5.354152] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131172
[ 5.354183] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131204
[ 5.354216] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131205
[ 5.354245] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131215
[ 5.354274] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131187
[ 5.354322] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131192
[ 5.354356] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131186
[ 5.354387] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131114
[ 5.354418] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131124
[ 5.354449] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131100
[ 5.354481] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131209
[ 5.354501] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131200
[ 5.354517] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131190
[ 5.354537] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131155
[ 5.354566] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131195
[ 5.354592] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131189
[ 5.354609] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131159
[ 5.354625] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131201
[ 5.354701] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131198
[ 5.354736] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131197
[ 5.354767] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131167
[ 5.354789] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131119
[ 5.354805] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131218
[ 5.354821] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131214
[ 5.354836] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131206
[ 5.354851] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131174
[ 5.354866] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131115
[ 5.354881] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131176
[ 5.354907] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131111
[ 5.354922] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131173
[ 5.354943] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131150
[ 5.354972] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131175
[ 5.355003] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131170
[ 5.355022] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131184
[ 5.355038] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131177
[ 5.355090] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131160
[ 5.355152] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131110
[ 5.355172] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131169
[ 5.355194] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131108
[ 5.355211] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131166
[ 5.355226] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131165
[ 5.355242] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131183
[ 5.355257] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131109
[ 5.355273] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131107
[ 5.355288] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131180
[ 5.355319] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131171
[ 5.355864] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131298
[ 5.355907] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131185
[ 5.355936] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131295
[ 5.355955] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131178
[ 5.356093] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131142
[ 5.356163] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131156
[ 5.356195] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131279
[ 5.356225] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131182
[ 5.356315] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131152
[ 5.356357] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131292
[ 5.356387] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131291
[ 5.356435] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131149
[ 5.356854] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131139
[ 5.356887] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131290
[ 5.356916] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131143
[ 5.356941] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131286
[ 5.356958] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131289
[ 5.356973] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131288
[ 5.356989] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131287
[ 5.357017] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131281
[ 5.357049] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131161
[ 5.357099] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131153
[ 5.357126] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131284
[ 5.357144] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131283
[ 5.357159] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131162
[ 5.357174] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131282
[ 5.357249] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131280
[ 5.357278] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131278
[ 5.357303] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131277
[ 5.357320] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131101
[ 5.357335] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131272
[ 5.357350] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131151
[ 5.357365] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131145
[ 5.357405] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131138
[ 5.357435] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131274
[ 5.357463] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131148
[ 5.357538] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131240
[ 5.357591] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131102
[ 5.357624] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131131
[ 5.357655] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131105
[ 5.357684] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131266
[ 5.357713] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131104
[ 5.357731] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131270
[ 5.357747] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131271
[ 5.357826] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131269
[ 5.357860] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131267
[ 5.357888] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131099
[ 5.357919] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131251
[ 5.357951] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131264
[ 5.357983] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 131260
[ 5.358009] EXT4-fs (vda1): 167 orphan inodes deleted
[ 5.361719] EXT4-fs (vda1): recovery complete
[ 5.375469] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
[ 9.465399] Adding 1045500k swap on /dev/vda5. Priority:-1 extents:1 across:1045500k
[ 9.514653] EXT4-fs (vda1): re-mounted. Opts: errors=remount-ro
以下を参考に、mysql_force_recovery=4をmy.cnfに追加して起動した。
http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
すると、今度は冒頭のログのあとに以下のような内容が出力されて起動できるようになった。
140113 20:13:13 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 1746989772
140113 20:13:13 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Warning: database page corruption or a failed
InnoDB: file read of space 0 page 5.
InnoDB: Trying to recover it from the doublewrite buffer.
InnoDB: Recovered the page from the doublewrite buffer.
InnoDB: Doing recovery: scanned up to log sequence number 1746989782
InnoDB: Last MySQL binlog file position 0 1112081, file name ./mysql-bin.000114
140113 20:13:14 InnoDB: Waiting for the background threads to start
140113 20:13:15 InnoDB: 5.5.32 started; log sequence number 1746989782
140113 20:13:15 InnoDB: !!! innodb_force_recovery is set to 4 !!!
140113 20:13:15 [Note] Recovering after a crash using mysql-bin
140113 20:13:15 [Note] Starting crash recovery...
140113 20:13:15 [Note] Crash recovery finished.
140113 20:13:15 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
140113 20:13:15 [Note] - '0.0.0.0' resolves to '0.0.0.0';140113 20:13:15 [Note] Server socket created on IP: '0.0.0.0'.
mysql_force_recoveryの設定をコメントアウトして再度起動するとリカバリは一応できているようだった。
slaveとして使っているものだったので、replicationを復帰させてリカバリ完了。