Sytem going to emergency mode when restarting SSHD / any service and asking for root password

Hello Team,

Please find the RCA for the issue Server carc-vhybp05 went in to maintenance mode
Server name : carc-vhybp05
Cause : Maintenance mode

Cause for the issue dm-6 device
Logs seen After resolving issue. i have found these errors after troubleshooting, observed device dm-6 was the cause for the issue as per below logs.

Mar 14 00:22:26 carc-vhybp05 kernel: [ 205.650511] XFS (dm-6): Invalid superblock magic number
Mar 14 00:22:26 carc-vhybp05 kernel: XFS (dm-6): Invalid superblock magic number [ ============>>>> /log which is device dm- 6 cause for the server maintenance ]
Mar 14 00:22:49 carc-vhybp05 kernel: [ 228.321470] EXT4-fs (dm-6): recovery complete
Mar 14 00:22:49 carc-vhybp05 kernel: EXT4-fs (dm-6): recovery complete
Mar 14 00:22:49 carc-vhybp05 kernel: EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
Mar 14 00:22:49 carc-vhybp05 kernel: [ 228.322736] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
Mar 14 00:25:52 carc-vhybp05 kernel: 411.195546] systemd-fstab-generator[14004]: Mount point is not a valid path, ignoring.
Mar 14 00:31:12 carc-vhybp05 kernel: Initializing cgroup subsys cpuset

I have uncommented the /log from /etc/fstab and rebooted the server to bring up, server came up after commenting /log in /etc/fstab

After commenting /log filesystem and once server came up seen below is the sequence of booting the devices up,
Here I can see dm-6 is not showing as we have commented in /etc/fstab .

Mar 14 00:31:15 carc-vhybp05 kernel: XFS (sda1): Mounting V4 Filesystem
Mar 14 00:31:16 carc-vhybp05 kernel: XFS (sda1): Ending clean mount
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-2): Mounting V4 Filesystem
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-3): Mounting V4 Filesystem
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-5): Mounting V4 Filesystem
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-4): Mounting V4 Filesystem
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-3): Ending clean mount
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-5): Ending clean mount
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-2): Ending clean mount
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-7): Mounting V5 Filesystem
Mar 14 00:31:17 carc-vhybp05 kernel: XFS (dm-7): Ending clean mount
Mar 14 00:31:18 carc-vhybp05 kernel: XFS (dm-4): Ending clean mount
Mar 14 00:31:18 carc-vhybp05 kernel: type=1305 audit(1521001878.305:3): audi

Mar 14 00:31:44 carc-vhybp05 kernel: [ 4.666920] Adding 2097148k swap on /dev/mapper/vg_root-lv_swap. Priority:-1 extents:1 across:2097148k FS
Mar 14 00:31:44 carc-vhybp05 kernel: [ 4.709909] XFS (sda1): Mounting V4 Filesystem
Mar 14 00:31:44 carc-vhybp05 kernel: [ 5.568137] XFS (sda1): Ending clean mount
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.396495] XFS (dm-2): Mounting V4 Filesystem
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.401301] XFS (dm-3): Mounting V4 Filesystem
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.403628] XFS (dm-5): Mounting V4 Filesystem
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.410001] XFS (dm-4): Mounting V4 Filesystem
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.452598] XFS (dm-3): Ending clean mount
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.454234] XFS (dm-5): Ending clean mount
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.466598] XFS (dm-2): Ending clean mount
Mar 14 00:31:44 carc-vhybp05 kernel: 6.488319] systemd-journald[521]: Received request to flush runtime journal from PID 1
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.601318] XFS (dm-7): Mounting V5 Filesystem
Mar 14 00:31:44 carc-vhybp05 kernel: [ 6.670950] XFS (dm-7): Ending clean mount
Mar 14 00:31:44 carc-vhybp05 kernel: [ 7.296935] XFS (dm-4): Ending clean mount
Mar 14 00:31:44 carc-vhybp05 kernel: [ 7.434090] type=1305 audit(1521001878.305:3): audit_pid=749 old=0 auid=4294967295 ses=4294967295 res=1
Mar 14 00:31:44 carc-vhybp05 kernel: [ 9.570701] NET: Registered protocol family 40
Mar 14 00:31:44 carc-vhybp05 kernel: [ 10.892682] vmxnet3 0000:0b:00.0 ens192: intr type 3, mo

After system came up, mounted /log filesystem manually and see the below logs, here /log mounted when mounted manually but it doesn’t mounted when system rebooted.
Here from the above first logs i have observed something wrong with filesystem.

Logs from manual mount
——————–

Mar 14 00:33:19 carc-vhybp05 kernel: [ 128.326128] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
Mar 14 00:33:19 carc-vhybp05 kernel: EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
Mar 14 00:53:01 carc-vhybp05 kernel: [ 1310.579444] device ens192 entered promiscuous mode
Mar 14 00:53:01 carc-vhybp05 kernel: device ens192 entered promiscuous mode

Resolution
================

Please find /etc/fstab output here /log is mounted with xfs filesystem as someone mounted it with xfs file system when server build or at some time as this is RHEL 7.4,
But when filesystem created it took dependency as EXT4 ( please find output from ” blkid ” command below)

[root@carc-vhybp05 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Aug 26 20:17:14 2016
i#
# Accessible filesystems, by reference, are maintained under ‘/dev/disk’
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_root-lv_root / xfs defaults 0 0
UUID=969966a3-51f2-4517-b394-e67edf9682bd /boot xfs defaults 0 0
/dev/mapper/vg_root-lv_home /home xfs defaults 0 0
/dev/mapper/vg_root-lv_opt /opt xfs defaults 0 0
/dev/mapper/vg_root-lv_tmp /tmp xfs defaults 0 0
/dev/mapper/vg_root-lv_var /var xfs defaults 0 0
/dev/mapper/vg_root-lv_swap swap swap defaults 0 0
/dev/datavg/datalv /data xfs defaults 0 0
/dev/mapper/vg_root-log_root /log xfs defaults 0 0
128.11.138.119:/ProductionRoot /production_nas nfs nolock 0 0
128.11.138.120:/StagingRoot /staging_nas nfs nolock 0 0

Output of blkid
===================

[root@carc-vhybp05 ~]# blkid
/dev/mapper/vg_root-lv_var: UUID=”f73233fd-f583-4bc3-9873-7914b3fd5e7d” TYPE=”xfs”
/dev/sda2: UUID=”ZQOzKa-BWjw-9s36-XCqL-LmaC-me6O-8OLWfS” TYPE=”LVM2_member”
/dev/mapper/vg_root-lv_root: UUID=”d5d8fd0b-326b-407d-92d5-10cf184245f0″ TYPE=”xfs”
/dev/sda1: UUID=”969966a3-51f2-4517-b394-e67edf9682bd” TYPE=”xfs”
/dev/mapper/vg_root-lv_swap: UUID=”a64425a2-fca6-4a6e-966c-fd3d20c7aea2″ TYPE=”swap”
/dev/sdb: UUID=”QWLVik-qLcf-0iAr-T3Ns-y8vz-Un9Q-OQ0CaC” TYPE=”LVM2_member”
/dev/mapper/vg_root-lv_home: UUID=”fb2f6bbe-78f8-4a85-82a6-03580dd25d15″ TYPE=”xfs”
/dev/mapper/vg_root-lv_opt: UUID=”0016a463-0453-4cc3-929e-1dc8bfcfc2cc” TYPE=”xfs”
/dev/mapper/vg_root-lv_tmp: UUID=”a9cfdefb-d128-4cce-86b6-406d2a9f439c” TYPE=”xfs”
/dev/mapper/vg_root-log_root: UUID=”f6a2f847-ee99-4fbe-adf1-c0d87c385c3e” TYPE=”ext4″
/dev/mapper/datavg-datalv: UUID=”d304d1bd-2dd6-4ac4-8c50-6395de4787d5″ TYPE=”xfs”

Output of df -h
=======================
[root@carc-vhybp05 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root 8.0G 5.1G 3.0G 64% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 12K 7.8G 1% /dev/shm
tmpfs 7.8G 8.8M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda1 497M 216M 281M 44% /boot
/dev/mapper/vg_root-lv_home 30G 1.8G 29G 6% /home
/dev/mapper/vg_root-lv_tmp 10G 33M 10G 1% /tmp
/dev/mapper/vg_root-lv_var 8.0G 1022M 7.0G 13% /var
/dev/mapper/datavg-datalv 50G 334M 50G 1% /data
/dev/mapper/vg_root-lv_opt 8.0G 277M 7.8G 4% /opt
tmpfs 1.6G 0 1.6G 0% /run/user/1025
128.11.138.120:/StagingRoot 250G 232G 19G 93% /staging_nas
128.11.138.119:/ProductionRoot 650G 625G 26G 97% /production_nas
tmpfs 1.6G 0 1.6G 0% /run/user/0
/dev/mapper/vg_root-log_root 13G 2.2G 9.9G 18% /log

Advertisements

Author: mylinuxissues

All the Issues posted here production resolved issues.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s