Subscribe to aSkDba.org Forum Posts

Enter your email address:

Delivered by FeedBurner


Askoracledba's Weblog Tech@aSkDbA.org


AskDBA.org Forum » Automatic Storage Management

ASM and Multipathing

(9 posts)
Average Rating For This Topic:

Rate This Topic Yourself:
  • Started 8 months ago by deepika.dhamija@gmail.com
  • Latest reply from deepika.dhamija@gmail.com
  • This topic is closed
  • This topic is resolved

  1. deepika.dhamija@gmail.com

    junior member
    Joined: Dec '09
    Posts: 9

    offline
    Reputation:12
     

    Hi

    I have a 2node RAC Setup 10gR2.

    I have three ASM Volumes configured, out of devices sdb2, sdb3, sdb4
    /etc/init.d/oracleasm createdisk VOL1 /dev/sdb2
    /etc/init.d/oracleasm createdisk VOL2 /dev/sdb3
    /etc/init.d/oracleasm createdisk VOL3 /dev/sdb4.
    where VOL1 & VOL2 belong to the Disk Group ORCL_DATA1 and Vol3 to the group FLASH_RECOVERY_AREA.

    I want to use the multipath disks namely /dev/multipath/sdb2, /dev/multipath/sdb3 and /dev/multipath/sdb4 for asm volumes.
    I created the multipath disks on node1 and edited the file /etc/sysconfig/oracleasm and set ORACLEASM_SCANORDER="multipath" and
    ORACLEASM_SCANEXCLUDE="sdb2 sdb3 sdb4".

    After a Reboot, the crs_stat shows
    On Node1

    --------------------------------------------------------------------------------
    Name Type Target State Host

    --------------------------------------------------------------------------------
    ora....SM1.asm application ONLINE ONLINE linux1
    ora....X1.lsnr application ONLINE ONLINE linux1
    ora.linux1.gsd application ONLINE ONLINE linux1
    ora.linux1.ons application ONLINE ONLINE linux1
    ora.linux1.vip application ONLINE ONLINE linux1
    ora....SM2.asm application ONLINE ONLINE linux2
    ora....X2.lsnr application ONLINE ONLINE linux2
    ora.linux2.gsd application ONLINE ONLINE linux2
    ora.linux2.ons application ONLINE ONLINE linux2
    ora.linux2.vip application ONLINE ONLINE linux2
    ora.orcl.db application ONLINE ONLINE linux1
    ora....l1.inst application ONLINE OFFLINE
    ora....l2.inst application ONLINE ONLINE linux2
    ora....test.cs application ONLINE ONLINE linux2
    ora....cl1.srv application ONLINE OFFLINE
    ora....cl2.srv application ONLINE ONLINE linux2

    oracle@linux1 ~$ sqlplus "/ as sysdba"

    SQL*Plus: Release 10.2.0.1.0 - Production on Tue Dec 8 12:04:53 2009

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    Connected to an idle instance.

    SQL> select name, path from v$asm_disk;
    select name, path from v$asm_disk
    *
    ERROR at line 1:
    ORA-01034: ORACLE not available

    I try to start the instance manually,
    oracle@linux1 ~$ srvctl start instance -d orcl -i orcl1
    PRKP-1001 : Error starting instance orcl1 on node linux1
    CRS-0215: Could not start resource 'ora.orcl.orcl1.inst'.

    Also from sql
    SQL> startup
    ORA-01078: failure in processing system parameters
    ORA-01565: error in identifying file '+ORCL_DATA1/orcl/spfileorcl.ora'
    ORA-17503: ksfdopn:2 Failed to open file +ORCL_DATA1/orcl/spfileorcl.ora
    ORA-15077: could not locate ASM instance serving a required diskgroup

    On Node2

    --------------------------------------------------------------------------------
    oracle@linux2 ~$ sqlplus "/ as sysdba"

    SQL*Plus: Release 10.2.0.1.0 - Production on Tue Dec 8 11:56:13 2009

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    Connected to:
    Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
    With the Partitioning, Real Application Clusters, OLAP and Data Mining options

    SQL> select name, path from v$asm_disk;

    NAME

    --------------------------------------------------------------------------------
    PATH

    --------------------------------------------------------------------------------
    VOL1
    ORCL:VOL1

    VOL2
    ORCL:VOL2

    VOL3
    ORCL:VOL3

    Plz note, that I have the multipath devices only on one node currently.

    I am new to the multipathing concept. Plz advise, on what (above) steps are wrong and what additional things are to be taken care of.
    And the causes/solutions to the errors.

    Posted 8 months ago #
  2. Amit Bansal

    Oracle DBA
    Joined: Jun '08
    Posts: 48

    offline
    Reputation:63
     

    Hi Deepika,

    I believe you are following below article available on Oracle website

    http://www.oracle.com/technology/tech/linux/asmlib/multipath.html

    First I believe that you should check ASM instance and see if you are able to see the disks on Node 1. From above output it looks that you connected to node 1 DB instance and not asm instance. This can also be seen from crs_stat output which shows that asm instance is up on node 1.
    Secondly name multipath device as multipatha,multibathc instead of /multipath/sdb1 etc

    Cheers
    Amit
    http://askdba.org/weblog/
    Posted 8 months ago #
  3. deepika.dhamija@gmail.com

    junior member
    Joined: Dec '09
    Posts: 9

    offline
    Reputation:12
     

    Hi Amit,

    Yes, I am following the above URL.

    On Node1:
    [root@linux1 ~]# /etc/init.d/oracleasm listdisks
    VOL1
    VOL2
    VOL3
    [root@linux1 ~]# /etc/init.d/oracleasm querydisk /dev/multipath/sdb2
    Device "/dev/multipath/sdb2" is marked an ASM disk with the label "VOL1"
    [root@linux1 ~]# /etc/init.d/oracleasm querydisk /dev/sdb2
    Device "/dev/sdb2" is marked an ASM disk with the label "VOL1"

    The O/P is same for sdb3 and sdb4 (both multipath & single path devices)
    Is this what you mean when you say if disks are visible on node1?

    [oracle@linux1 ~]$ srvctl status asm -n linux1
    ASM instance +ASM1 is running on node linux1.
    [oracle@linux1 ~]$ srvctl status instance -d orcl -i orcl1
    Instance orcl1 is not running on node linux1
    [oracle@linux1 ~]$ srvctl start instance -d orcl -i orcl1
    PRKP-1001 : Error starting instance orcl1 on node linux1
    CRS-0215: Could not start resource 'ora.orcl.orcl1.inst'.
    [oracle@linux1 ~]$

    I am unable to start the instance orcl1 on node1.

    Also what is the significance of naming multipath devices in a particular way?

    Posted 8 months ago #
  4. Amit Bansal

    Oracle DBA
    Joined: Jun '08
    Posts: 48

    offline
    Reputation:63
     

    Hi,

    Try logging into ASM instance and query V$ASM_DISK i.e

    export ORACLE_SID=+ASM1
    sqlplus "/ as sysdba"
    select name,path,mount_status,header_status from V$ASM_DISK;

    with regards to naming convention, I think naming convention should follow /dev/[a-z]* naming convention instead of /dev/multipath/[a-z]*
    It could be this or it could be that as the name of multipath device is also ending with sdb2,sdb3 and we are also excluding the disk ending with sdb2,sdb3. I am not too sure on this.

    I think best way would be to actually first query V$ASM_DISK output.

    Posted 8 months ago #
  5. deepika.dhamija@gmail.com

    junior member
    Joined: Dec '09
    Posts: 9

    offline
    Reputation:12
     

    Thanks Amit for the prompt reply.

    I had ORACLE_SID SET to orcl1 and orcl2 on both the nodes respectively.

    On node1, I did the above steps, and I luckily was able to connect to Oracle. It did not say 'connected to idle instance'.

    The O/P of the query on Node1 (ORACLE_SID=+ASM1) is
    NAME PATH MOUNT_S HEADER_STATUS
    ORCL:VOL1 CLOSED UNKNOWN
    ORCL:VOL1 CLOSED UNKNOWN
    ORCL:VOL1 CLOSED UNKNOWN

    The O/P of the query on Node2 (ORACLE_SID=orcl2) is
    NAME PATH MOUNT_S HEADER_STATUS
    VOL1 ORCL:VOL1 OPENED UNKNOWN
    VOL2 ORCL:VOL2 OPENED UNKNOWN
    VOL3 ORCL:VOL3 OPENED UNKNOWN

    Does that mean that the multipath disks have not been picked up by asm at all.
    What about the permissions. I ran the foll commands (for sdb2, sdb3, sdb4) on Node1
    # chown -R oracle:dba /dev/multipath/sdb2
    # chmod -R 775 /dev/multipath/sdb2

    # /etc/init.d/oracleasm scandisks
    # /etc/init.d/oracleasm listdisks
    VOL1
    VOL2
    VOL3

    How do these permissions remain intact after a reboot to the node?

    Posted 8 months ago #
  6. Amit Bansal

    Oracle DBA
    Joined: Jun '08
    Posts: 48

    offline
    Reputation:63
     

    Deepika,

    How have you configured multipathing? Are you using udev and device mapper?
    Also let me know your linux release details.

    -Amit

    Posted 8 months ago #
  7. deepika.dhamija@gmail.com

    junior member
    Joined: Dec '09
    Posts: 9

    offline
    Reputation:12
     

    Hi Amit,

    Actually I have an application that runs on every node on the cluster and creates the devices /dev/multipath/sd*. I am new to this entire setup, but I can tell you that it uses udev.
    On Node 1
    [oracle@linux1 ~]$ uname -a
    Linux linux1 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 7 17:52:47 EST 2009 i686 i686 i386 GNU/Linux

    Does this much information answer your questions?

    Posted 8 months ago #
  8. Amit Bansal

    Oracle DBA
    Joined: Jun '08
    Posts: 48

    offline
    Reputation:63
     

    Hi,

    >>Actually I have an application that runs on every node on the cluster and creates the devices /dev/multipath/sd*

    Can't understand this statement.
    Anyways if you use udev, you can set permissions in rules file

    /etc/udev/rules.d/50-udev.rules

    I think your sysadmin can help you with that.

    -Amit

    Posted 8 months ago #
  9. deepika.dhamija@gmail.com

    junior member
    Joined: Dec '09
    Posts: 9

    offline
    Reputation:12
     

    Amit,

    I am finally able to achieve what I intended to do.

    Many Thanks to you for your valuable suggestions.

    Posted 8 months ago #

RSS feed for this topic

Topic Closed

This topic has been closed to new replies.

158 posts in 58 topics over 28 months by 43 of 83 members. Latest: shyamsundar, oraclebychoice, taraldesai