Tuesday night (12/02/2008), one harddisk on Toyota’s server going down. That disk located in the mirror which build using Veritas Volume Manager. Toyota is my office’s biggest & carping customer. So many people in my office going busy with this accident. From Tuesday night I’m as support engineer have made the manual procedure for replacing the failure disk. But guess what? They prefer to ask the Veritas engineer rather than follow my procedure. So why bothering me call me and report the problem? Why they didn’t ask the expert only? I think engineer like me only used for physical work beside troblesho0ting things.

Maybe all the manager in my office get this lesson : “don’t trust your engineer, ask the expert only”.

Finally, we replaced the broken disk at Wednesday night. With all eyes on me, I executed the replacing procedures. The broken disk is the rootmirror disk in SunFire V890. The procedure for replace rootmirror disk which hold by Veritas Volume Manager (VxVM) is like this :

1. Check the status of all disks that hold by Veritas Volume Manager using :
   # vxdisk list
Using this command for locate which disk that failed. See the following illustration :
2. Check whether the failed disk can be remove from the operating system (in Toyota case, the OS is Solaris 9/04) :
   # luxadm remove_device /dev/rdsk/c1t2d0s2
   If that command cannot be executed, you may just pull out the disk from the server. After that you must replace the disk with the new one as soon as possible.
3. Run devfsadm and cfgadm -c configure c2 from Solaris OS to let the system know the new disk. In Toyota case, the failed disk located at c1t2d0. See the following illustration before and after the OS knowing the new disk :
4. Try to see all the disk using format command. If the new disk has been appears, you must label the disk.
5. Run the vxdctl, this is the command for let the VxVM knows all the devices that connected to the server (including the new attached devices).
6. Run the vxdiskadm (disk configurations menu of VxVM), using this command :
   # vxdiskadm
See the following illustration :
7. Choose the fourth menu in the vxdiskadm which is "Remove a disk for replacement". Choose this menu by type “4” in the vxdiskadm menu. Program will see the failed disk that has been remove from VxVM menu. Choose that disk for removal process. In Toyota case, the failed disk was rootmirror01. See the following illustration :
8. After that choose the fifth menu which is "Replace a failed or removed disk". Choose this menu by type “5” in the vxdiskadm menu. By running this program Veritas will know that servers has one new disk that haven’t assign to the Veritas Volume Manager. In Toyota case, VxVM knows that c1t2d0 is the new disk that hasn’t assign into VxVM. They will ask you, whether you want to assign the c2t1d0 to be rootmirror01. When VxVM ask you to encapsulate disk, you must reject that. Instead of encapsulate the new disk, you must answer yes when VxVM offer you to initialize the new disk. See the following illustration :
9. Follow the rest of the vxdiskadm menu, don’t worry it’s so easy. After that quit the menu. The new disk has been added into VxVM configuration and they start the synchronization between rootmirror01 with the rootdisk01.  Compare the list of all disks after disk replacement :
   
10. The synchronization process occured as long 2 hours for 146 GB disks. We can check the progress of synchronization process using this command :
    # vxtask list
    See the following example :

Here the answer :

Setting up OpenSUSE 10.3 as DHCP server and Internet gateway 😀 Complete story will post soon.

Before :

After :

Entah maksudnya apa papan petunjuk Kantor Pos ini ditulis terbalik :

Keisengan PT Pos atau salah cetak tulisan di papan petunjuk. Gambar tadi saya ambil ketika melintas di daerah Pondok Labu, lupa apa nama jalannya.

Malam ini saya dan rekan-rekan di kos makan-makan di Pulau Dua restoran. Pulau Dua ada di Jalan Gatot Subroto, berada di dekat gedung DPR MPR. Ini foto saya setelah menghabiskan banyak makanan malam ini ;)) :

Kami berenam ceritanya ditraktir Edi rekan saya, katanya Edi baru dapat project besar..(he..he..he..thanks to Edi, good luck with your next project). Berenam kami pesan sup kepiting hisit asparagus, kepiting saus pedas, ayam goreng khas Pulau Dua, udang bakar, udang asam manis, gurami goreng kering, kangkung cah sapi, buncis cah (lupa dimasak apa buncisnya). Pelayannya senyum-senyum saja saat kami memesan begitu banyak makanan. Saking herannya sampai si mas pelayan tadi bertanya : ini mau porsi besar semua? 😀 mungkin mas pelayan tadi mikir ini orang kelaparan atau kesurupan =)) Foto di atas diambil saat kami sudah selesai makan, piring saya penuh kulit kepiting & udang; lihat di sampingnya : ada 1 piring yang juga penuh sisa makanan saya :-p

Pulau Dua tempatnya memang asik untuk kumpul-kumpul. Saung-saung yang menghadap danau cukup menyenangkan kalau dijadikan tempat ngobrol. Makanan di sana sebenarnya tidak terlalu spesial. Dari sekian menu yang disajikan yang menurut saya enak adalah ayam goreng khas Pulau Dua dan udang bakarnya. Yang tidak enak adalah kangkung cah sapi….datar sekali rasanya. Minuman yang kami pesan pun tidak seperti yang diharapkan. Pesan fresh orange juice dan jus jambu, minuman yang datang kok rasanya sepertinya jus botolan merek Berry. Eh tapi Berry juga jus asli kan ya?

Jakarta masih diguyur hujan terus sedari sore tadi. Saya datang ke rumah terlambat. Pulang dari Siemens jam hampir setengah tujuh. Setelah mengantarkan rekan kantor saya dulu, Bluebird yang saya tumpangi saya suruh ngebut sampai rumah. Hujan makin deras. Rencana awal kami semua mau pergi makan malam pukul setengah 8. Eh akhirnya baru jam 8 kami berangkat. Tidak semua rekan kos bisa ikut makan-makan. Ada yang belum pulang dari kantor, ada juga yang ditelepon bosnya untuk ikutan meeting mendadak. Padahal awalnya kami sudah siapkan rencana ini jauh-jauh hari sebelum hari ini. Ya memang sudah mengumpulkan banyak orang apalagi yang semuanya punya kesibukan masing-masing. Dari sore saya sudah lapar, jadi harap maklum kalau malam ini saya makan banyak :-p Once again, thanks Mr Edi for your treat.