DRBD on Red Hat Enterprise Linux 5

This is a rough guide and really for future personal references so I can rebuild a DRBD cluster in 3 minutes, without having to dig into DRBD docs again.
Please correct me if something is wrong (I’m thinking about DRBD gurus, Arrfab coming to mind :-) ) .

This worked for me on RHEL5.6 64 bits and DRBD 8.2.

I’ll call the DRBD resource “DRBDCluster”

Install drbd8X and kmod-drbd8X (grab the RPMS at your favorite RPM retailer.. Seems like http://elrepo.org/linux/elrepo/el5/ a good place to start, thought I haven’t tried their RPMS)

On both node, create /etc/drbd.conf with something like this :

global {
    usage-count yes;
}
common {
    syncer {
        rate 1M;
    }
}

resource DRBDCluster {
    protocol C;
    net {
        cram-hmac-alg sha1;
        shared-secret "sdfdskljdfklfjsdkljfsdkljflk";
    }
    on srv1.intranet.example.org {
        device    /dev/drbd0;
        disk      /dev/sdb1;
        address   1.1.1.1:7789;
        meta-disk internal;
    }
    on srv2.intranet.example.org {
        device    /dev/drbd0;
        disk      /dev/sdb1;
        address   1.1.1.2:7789;
        meta-disk internal;
    }
}

On the “master” (srv1) :

- create the partition found in config (DO NOT FORMAT) :
fdisk /dev/sdb1
- Then create the actual DRBD device :
drbdadm create-md DRBDCluster
- if not already done :
modprobe drbd
- start DRBD. It will complain that it can’t “see” the other node, type “yes” at “To abort waiting enter ‘yes’”. :
service drbd start
- set the node as primary :
drbdadm -- --overwrite-data-of-peer primary DRBDCluster
- format /dev/drbd0 :
mkfs.ext3 /dev/drbd0

On the slave node (srv2) :
service drbd start

Now you can check the status of the DRBD cluster.

It’s sync’ing.

srv2# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
ns:0 nr:6016 dw:6016 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:10476008
[>....................] sync'ed: 0.2% (10230/10236)M
finish: 1:49:07 speed: 1,504 (1,504) K/sec

The “oos” flag expresses the amount of data to be synced, in KB.

When the sync is done it will look like this :

srv1# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:10482024 nr:0 dw:298456 dr:10482072 al:121 bm:640 lo:0 pe:0 ua:0 ap:0 oos:0

If for some reason the slave node becomes unavailable, it will resync what it has missed when it comes back (here 20 MB of data) :

version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:10486540 nr:0 dw:319096 dr:10486693 al:130 bm:642 lo:0 pe:0 ua:0 ap:0 oos:16124
[=========>..........] sync'ed: 50.0% (16124/20640)K
finish: 0:00:10 speed: 1,504 (1,504) K/sec

Useful commands :

Set the sync speed on a running DRBD cluster :
drbdsetup /dev/drbd0 syncer -r 10M

There’s plenty of discussion on the right speed to set. It really all depends on your application, network, etc.

More useful commands to come later.

I have integrated DRBD into a RHCS Cluster (for Zarafa), check the doc here : http://www.drbd.org/users-guide/ch-rhcs.html

14 thoughts on “DRBD on Red Hat Enterprise Linux 5

  1. Pingback: Tweets that mention Sébastien Wains » DRBD on Red Hat Enterprise Linux 5 -- Topsy.com

    • There’s no exact science with this setting. It really all depends on the application, how much data has to be replicated, if the nodes are on the same network/premises or not.

      The default value used to slow down a Zarafa Cluster. HP G4 nodes were two separate physical locations distant from around 5 kms, with a Gigabit link.

      I’d say that this setting has to be worked out during the test phase. Decrease the value until the application is responsive in high load tests.

  2. Hey:) your posts are great! not only about OpenVpn:)

    And yeah it rocks. no downtime – no problems with dynamical ips, I love it:)

  3. Sébastien what was your solution to access zarafa web-acces? I mean you have 2 separated servers with different locations. Have you a special solution for the virtual-IP?
    _ _ _ _ _ _ _ _ _
    srv1.intranet.example.org has a network A – 7.7.7.11
    srv2.intranet.example.org has a network B – 8.8.8.10
    What I mean you can’ t have a virtual IP with different networks?
    - – - – - – - – - – - – -

    I think you have a virtual IP-Address to access the running server, but what was the solution to bring it to work with different networks and a virtual IP?

    By the way what is your cluster stack?

    Thank you!

    • For the gateway side (MTA, webmail, IMAPS, POP3S) we are using a Cisco ACE load balancer doing SSL offload. With this solution it is possible to load balance between 2 separate networks, as long as the load balancer can reach those two real servers.

      For the Zarafa side, we are using Cisco ACE along RHCS in failover mode. Cisco ACE VIP is publicly available, RHCS VIP is internally available only. Cisco ACE is somewhat doing DNAT here.

      I guess srv1 and srv2 are internal networks and you want to make the webmail publicly accessible, right ? But do you want to load balance or failover ?

  4. I’m working currently on a failover solution – for 2 servers with different locations. That means I have a Problem with my virtual IP. It will be a web-server with cluster filesystem (can’t decide which one:) – maybe you can recommend something? OCFS2?) to run mysql and apache on each node under DRBD.

    Example:
    http://www.example.com –> vhosts.example.com
    vhosts.example.com –> srv1.example.com and srv2.example.com

    If one server goes down, everything should be taken by another node. My provider (hetzner.de) sells Failover-IP which can be variable assigned to a server. This is the only one solution I see.

    Thank you for a quick reply.

    • For the MySQL part, we have great success running it over DRBD. I would stick to this solution personally.
      I have made some tests with a failover Apache cluster over RHCS and DRBD. It works fine and easy to set up.
      I tend to avoid anything involving Oracle, never played with OCFS2.

      Take a look at IPVS and Keepalived. I haven’t played with those in years, but I believe it should fit. Let me know if it does, or if I’m completely mistaken :-)

  5. hi wains,

    i receive this error. what does it mean.

    Found LVM2 physical volume signature
    Device size would be truncated, which
    would corrupt data and result in
    ‘access beyond end of device’ errors.
    If you want me to do this, you need to zero out the first part
    of the device (destroy the content).
    You should be very sure that you mean it.
    Operation refused.

    thanks
    nazrin

  6. hi wains,

    i have figure out the problem for above comment. now i got issue when try to start for drbd services…

    [root@primary testsvr]# /etc/rc.d/init.d/drbd start
    Starting DRBD resources: [ d(repdata) /dev/drbd0: Failure: (114) Lower device is already claimed. This usually means it is mounted.

    [repdata] cmd /sbin/drbdsetup /dev/drbd0 disk /dev/sda /dev/sda internal –set-defaults –create-device –on-io-error=detach failed – continuing!

    ].

Comments are closed.