ESXi - removing unknown nfs volume

esxi에서 mount 중인 nfs 볼륨의 타겟 스토리지의 문제로 미쳐 볼륨을 umount를 하지 못한 경우,

datastore에서 해당 볼륨을 삭제하려고 하면 에러와 함께 삭제가 되지 않는다.

그럴 때는 아래와 같은 방법으로 삭제하면 된다.

 

1. esxi에 ssh로 접속후 볼륨 리스트 확인

[root@localhost:~] ls -la /vmfs/volumes/
ls: /vmfs/volumes/235b504b-9a93af38: Input/output error
total 7944
drwxr-xr-x    1 root     root           512 May 15 13:57 .
drwxr-xr-x    1 root     root           512 Apr 24 08:09 ..
drwxr-xr-x    1 root     root             8 Jan  1  1970 020390e5-da55291d-0f98-4363088643ce
drwxr-xr-t    1 root     root          1960 Apr  9 12:43 589c11c7-07322a7b-cd64-3ca315048dc2
drwxr-xr-t    1 root     root         81920 Apr 12 07:11 5b917a52-f0a659d0-b62d-7085c284de0e
drwxr-xr-t    1 root     root         77824 Apr 12 07:09 5ba0e754-c24a956e-1e37-6805ca2db245
drwxr-xr-x    1 root     root             8 Jan  1  1970 5bb32822-15b5cd6e-7402-7085c284de0e
drwxr-xr-t    1 root     root         77824 May 12 03:56 5c255f39-bee3047e-f6ba-3ca315048dfe
drwxr-xr-t    1 root     root         73728 Aug 25  2019 5d6273ea-2d2a5c54-f7e3-64e599f46e7e
drwxr-xr-t    1 root     root         73728 Aug 25  2019 5d6273fe-542df4f2-46a2-64e599f46e7e
drwxr-xr-t    1 root     root         73728 Aug 25  2019 5d62740b-aac5704c-f513-64e599f46e7e
lrwxr-xr-x    1 root     root            35 May 15 13:57 HDD01 -> 589c11c7-07322a7b-cd64-3ca315048dc2
lrwxr-xr-x    1 root     root            35 May 15 13:57 HDD02 -> 5c255f39-bee3047e-f6ba-3ca315048dfe
lrwxr-xr-x    1 root     root            35 May 15 13:57 HDD03 -> 5b917a52-f0a659d0-b62d-7085c284de0e
lrwxr-xr-x    1 root     root            17 May 15 13:57 NFS01 -> ffc34b8a-76ba43a8
lrwxr-xr-x    1 root     root            17 May 15 13:57 NFS02 -> 235b504b-9a93af38    ---> 문제의 nfs 볼륨 (235b504b-9a93af38 디렉토리가 없고, unlink 상태이다.)
lrwxr-xr-x    1 root     root            17 May 15 13:57 NFS03 -> b2140c5d-8c8b741e
lrwxr-xr-x    1 root     root            35 May 15 13:57 NVME01 -> 5ba0e754-c24a956e-1e37-6805ca2db245
lrwxr-xr-x    1 root     root            35 May 15 13:57 XNAS01 -> 5d6273ea-2d2a5c54-f7e3-64e599f46e7e
lrwxr-xr-x    1 root     root            35 May 15 13:57 XNAS02 -> 5d6273fe-542df4f2-46a2-64e599f46e7e
lrwxr-xr-x    1 root     root            35 May 15 13:57 XNAS03 -> 5d62740b-aac5704c-f513-64e599f46e7e
drwxr-xr-x    1 root     root             8 Jan  1  1970 aa4754e9-037afeb0-06b9-940dc1bcfbf5
drwxrwxrwx    4 root     root            31 May 15 13:30 b2140c5d-8c8b741e
drwxrwxrwx    4 root     root          4096 May 12 04:53 ffc34b8a-76ba43a8

 

2. 링크 삭제 시도

[root@localhost:~] rm -f /vmfs/volumes/NFS02
rm: can't remove '/vmfs/volumes/NFS02': Function not implemented

삭제되지 않는다.

 

3. 볼륨 리스트 확인

[root@localhost:~] esxcli storage nfs list
Volume Name  Host           Share                                 Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
-----------  -------------  ------------------------------------  ----------  -------  ---------  -----  ---------------------
NFS03        192.168.1.190  /home/mapoo/mapoo7/osbackup/vmbackup        true     true      false  false  Not Supported
NFS02        192.168.0.250  /home/vm_backup                            false     true      false  false  Unknown         ---> 문제의 nfs 볼륨
NFS01        192.168.0.250  /data0/mapoo/mapoo7/nfs-share               true     true      false  false  Not Supported

--> 예를 들어 만약 NFS02 볼륨 연결을 끊기 전에 서버측 IP가 변경되어 버린 경우,

다시 NFS02 라는 이름을 바뀐 IP로 연결을 시도하면 이미 이름이 있다면서 연결이 되지 않는다. 그리고 nfs list에도 NFS02는 확인되지 않는다.

그럴 때에서 아래 5. 번 내용대로 nfs remove로 제거하고 다시 같은 이름으로 연결을 하면된다.

 

4. storageRM 중지/Rescan/시작

[root@localhost:~] /etc/init.d/storageRM stop
watchdog-storageRM: Terminating watchdog process with PID 2098633
storageRM stopped

--> ESXi web console에서 host를 Rescan 한다.

[root@localhost:~] /etc/init.d/storageRM start
storageRM started

이 스텝은 생략해도 괜찮다.

 

5. nfs 볼륨 삭제

[root@localhost:~] esxcli storage nfs remove -v NFS02

 

6. 삭제 확인

[root@localhost:~] ls -la /vmfs/volumes/
total 7944
drwxr-xr-x    1 root     root           512 May 15 14:19 .
drwxr-xr-x    1 root     root           512 Apr 24 08:09 ..
drwxr-xr-x    1 root     root             8 Jan  1  1970 020390e5-da55291d-0f98-4363088643ce
drwxr-xr-t    1 root     root          1960 Apr  9 12:43 589c11c7-07322a7b-cd64-3ca315048dc2
drwxr-xr-t    1 root     root         81920 Apr 12 07:11 5b917a52-f0a659d0-b62d-7085c284de0e
drwxr-xr-t    1 root     root         77824 Apr 12 07:09 5ba0e754-c24a956e-1e37-6805ca2db245
drwxr-xr-x    1 root     root             8 Jan  1  1970 5bb32822-15b5cd6e-7402-7085c284de0e
drwxr-xr-t    1 root     root         77824 May 12 03:56 5c255f39-bee3047e-f6ba-3ca315048dfe
drwxr-xr-t    1 root     root         73728 Aug 25  2019 5d6273ea-2d2a5c54-f7e3-64e599f46e7e
drwxr-xr-t    1 root     root         73728 Aug 25  2019 5d6273fe-542df4f2-46a2-64e599f46e7e
drwxr-xr-t    1 root     root         73728 Aug 25  2019 5d62740b-aac5704c-f513-64e599f46e7e
lrwxr-xr-x    1 root     root            35 May 15 14:19 HDD01 -> 589c11c7-07322a7b-cd64-3ca315048dc2
lrwxr-xr-x    1 root     root            35 May 15 14:19 HDD02 -> 5c255f39-bee3047e-f6ba-3ca315048dfe
lrwxr-xr-x    1 root     root            35 May 15 14:19 HDD03 -> 5b917a52-f0a659d0-b62d-7085c284de0e
lrwxr-xr-x    1 root     root            17 May 15 14:19 NFS01 -> ffc34b8a-76ba43a8
lrwxr-xr-x    1 root     root            17 May 15 14:19 NFS03 -> b2140c5d-8c8b741e
lrwxr-xr-x    1 root     root            35 May 15 14:19 NVME01 -> 5ba0e754-c24a956e-1e37-6805ca2db245
lrwxr-xr-x    1 root     root            35 May 15 14:19 XNAS01 -> 5d6273ea-2d2a5c54-f7e3-64e599f46e7e
lrwxr-xr-x    1 root     root            35 May 15 14:19 XNAS02 -> 5d6273fe-542df4f2-46a2-64e599f46e7e
lrwxr-xr-x    1 root     root            35 May 15 14:19 XNAS03 -> 5d62740b-aac5704c-f513-64e599f46e7e
drwxr-xr-x    1 root     root             8 Jan  1  1970 aa4754e9-037afeb0-06b9-940dc1bcfbf5
drwxrwxrwx    4 root     root            31 May 15 13:30 b2140c5d-8c8b741e
drwxrwxrwx    4 root     root          4096 May 12 04:53 ffc34b8a-76ba43a8

esxi web console의 datastore 리스트에서도 NFS02 볼륨이 사라졌을 것이다.