http://vipul.kothiyal.com/blog/

and will delete this one soon.
Cheers
Vipul

error messages in syslog:

NETDEV WATCHDOG: eth2: transmit timed out
tg3: eth2: transmit timed out, resetting
tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=3400 enable_bit=2

After some googling it seemed to be  a known issue and way out was to a) Disable flow control and

b) update the driver

To disable flow control all you need to do is, at command prompt:

ethtool -K ethX(interface number) tso off

Now the next bit to compile the driver for centos 4

1) Get the driver srpm (Yes I like RPMs, I just do) from broadcom ot HP website

2) On the server make sure you have Kernel source, gcc, rpm-build installed

basically

yum install rpm-build gcc kernel-devel

on RHEL4 you’ll need kernel-headers as well

go to directory where you have kept the driver SRPM and do as below:

rpm -ivh driver.src.rpm

(if you get error for unable to create SOURCES directory then create redhat directory in /usr/src)

now cd /usr/src/redhat

rpmbuild -bb SPECS/tg3.spec

The rpm will get built in RPMS directory in /usr/src/redhat go in there and install it  as below

rpm -Uvh tg3-3.110g-1().i386.rpm

you can use this rpm on any other machine which has same kernel and off course the said hardware.

Now a reboot is needed.

Cheers

Add this line in your run.sh for Jboss:

JAVA_OPTS=”$JAVA_OPTS  -Djava.rmi.server.hostname=`hostname` -Dcom.sun.management.jmxremote.port=9001 Dcom.sun.management.jmxremote.ssl=false \    -Dcom.sun.management.jmxremote.authenticate=false”

notice -Djava.rmi.server.hostname=`hostname` , this bit is not in standard docs however this is what made mine work as it seems without it JMX could not bint to the hostname, strange yes I know, but hey we live in a  strange world

Update: There is another caviat with JMX remote if you have iptables enabled on your server.

Basically when you enable JMX remote it creates 2 sockets, one on the port specified and the other one it creates on a random port, each time you restart Jboss this port changes, I am not sure what it is for but it should be open for you to be able to connect JMX remote.
Now because it is a random port we can’t add a rule in iptables for it, hence no JMX remote.
There could be 3 options to go about it:
a) You dig the bit that generates a random port and hardcode it to a fixed port or some how enable some config to assign it to a fixed port . (Best pref)

b) Each time after you restart Jboss yo do this:
i) stop iptables /etc/init.d/iptables stop
ii) connect to jconsole from your machine and not down the dynamic port by doing

netstat -a | grep {name of you machine}
iii) open this port in iptables and start iptables again

(Ugly hack)

c) disable iptabels (I wouldn’t prefer this)

We had an old Sun Server(SunV240) which needed external(more basically) storage. Our vendor suggested us to go with Sun Storagetek 2530 based on our space requirements. Now the fun started as we were delivered with a wrong card(PCIe) to connect the array with, which obviously was a stupid mistake to make when you do this work day in day out unlike me., Well after some haggling by one of my colleagues they delivered the correct card and I did the connection. Here is what I did:
1) Connect both the controllers of 2530 to Management LAN (As the only way you'll be able to configure/manage 2530 is via Common Array Manager, which if you don't already have installed then you'll have  to install it on a host, this can be Linux(yay!!), widows or solaris(no way)).
2) Once you  have the array connected to network and it has received it's IP address from your DHCP server(Yup you are in for trouble if you don't have a dhcp server or you like to assign static IPs to management LAN interfaces), then you'll need to add this array to you CAM host, which is not very painful, (In my setup I did notice that our DHCP server didn't assign a default gateway to these interfaces)
3) Now that that Array is added in to CAM host you need to make sure/do following on the array:
■  The system time has beencorrectly set.
■  At least one new user has been added and assigned the roleof “storage.”
■  An initiator has been created.
■  At least one host has been created and mapped to an initiator.
■  At least one host group has been created.
■  A storage pool has been created.
■  A volume has been created and mapped to a host or host group.
It's really straight forward to accomplish these Via CAM web GUI.
4) Now log on to the server you have connected the array to and switch user to super user and do following
#drvconfig
#disks
you can check if your array has been detected or not
#  ls -lt /dev/rdsk/ | more
total 64
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s7 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:h,raw
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s6 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:g,raw
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s5 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:f,raw
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s4 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:e,raw
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s3 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:d,raw
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s2 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:c,raw
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s1 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:b,raw
lrwxrwxrwx   1 root     root          55 Mar  1 13:53 c2t0d0s0 -> ../../devices/pci@1d,700000/LSILogic,sas@1/sd@0,0:a,raw
Now all we need to do is partition the new drive so do following:
#format
you'll see something like;
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1c,600000/scsi@2/sd@0,0
1. c0t1d0
/pci@1c,600000/scsi@2/sd@1,0
2. c2t0d0
/pci@1d,700000/LSILogic,sas@1/sd@0,0
Specify disk (enter its number):
Ours was disk number 2, so that's what I entered and it'll show somehting like:
selecting c2t0d0
[disk formatted]
FORMAT MENU:
disk       - select a disk
type       - select (define) a disk type
partition  - select (define) a partition table
current    - describe the current disk
format     - format and analyze the disk
repair     - repair a defective sector
label      - write label to the disk
analyze    - surface analysis
defect     - defect list management
backup     - search for backup labels
verify     - read and display labels
save       - save new disk/partition definitions
inquiry    - show vendor, product and revision
volname    - set 8-character volume name
!     - execute , then return
quit
format>
Enter partition on at format prompt and you shall see something like
format> partition
PARTITION MENU:
0      - change `0' partition
1      - change `1' partition
2      - change `2' partition
3      - change `3' partition
4      - change `4' partition
5      - change `5' partition
6      - change `6' partition
7      - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name   - name the current table
print  - display the current table
label  - write partition map and label to the disk
!<cmd> - execute <cmd>, then return
quit
partition>
Enter p and it'll show you current drive partition table which will have root, swap, backup and usr partitons which we don't need as this is just going to be used as data disk so we get rid of them by doing:
on partion prompt enter the partition number
partition> 0
Part      Tag    Flag     Cylinders        Size            Blocks
  0       root    wm       0 -   55      128.84MB    (56/0/0)     263872
Enter partition id tag[root]: unassigned
Enter partition permission flags[wm]:
Enter new starting cyl[0]:
Enter partition size[263872b, 56c, 128.84mb, 0.13gb]: 0
partition>
do the same with swap, backup and usr.
now that you have removed the default partitions, it's time to create new  and to do that we need to do as below.
primarily what we 'll do is :
1) modify
2) label
3) quit the format utility
4) create file system and edit /etc/vfstab
Here are the actual commands:
partition> 1       
Part      Tag    Flag     Cylinders         Size            Blocks
 1 unassigned    wm       0 -  8702      200.01GB    (8703/0/0)   419441085
Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 8703
Enter partition size[419441085b, 8703c, 17405e, 204805.22mb, 200.01gb]:
partition> p
Current partition table (unnamed):
Total disk cylinders available: 48541 + 2 (reserved cylinders)
Part      Tag    Flag     Cylinders         Size            Blocks
 0 unassigned    wm       0 -  8702      200.01GB    (8703/0/0)   419441085
 1 unassigned    wm    8703 - 17405      200.01GB    (8703/0/0)   419441085
Please notice the cylinder number, don't overlap them.So now the next slice(partition) will be start at cylinder 17406.
I needed to create 5 slices of 200g each so at after repeating the above step for all my partiton table looked like:
partition> p
Current partition table (unnamed):
Total disk cylinders available: 48541 + 2 (reserved cylinders)
Part      Tag    Flag     Cylinders         Size            Blocks
 0 unassigned    wm       0 -  8702      200.01GB    (8703/0/0) 419441085
 1 unassigned    wm    8703 - 17405      200.01GB    (8703/0/0) 419441085
 2 unassigned    wm   17406 - 26108      200.01GB    (8703/0/0) 419441085
 3 unassigned    wm   26109 - 34811      200.01GB    (8703/0/0) 419441085
 4 unassigned    wm   34812 - 43514      200.01GB    (8703/0/0) 419441085
 5 unassigned    wm       0                0         (0/0/0)            0
 6 unassigned    wm       0                0         (0/0/0)            0
 7 unassigned    wm       0                0         (0/0/0)            0
afetr this do:
partition> mod
Select partitioning base:
 0. Current partition table (unnamed)
 1. All Free Hog
Choose base (enter number) [0]?
Warning: Non-contiguous partition (3) in table.
Warning: Non-contiguous partition (4) in table.
Part      Tag    Flag     Cylinders         Size            Blocks
 0 unassigned    wm       0 -  8702      200.01GB    (8703/0/0) 419441085
 1 unassigned    wm    8703 - 17405      200.01GB    (8703/0/0) 419441085
 2 unassigned    wm   17406 - 26108      200.01GB    (8703/0/0) 419441085
 3 unassigned    wm   26109 - 34811      200.01GB    (8703/0/0) 419441085
 4 unassigned    wm   34812 - 43514      200.01GB    (8703/0/0) 419441085
 5 unassigned    wm       0                0         (0/0/0)            0
 6 unassigned    wm       0                0         (0/0/0)            0
 7 unassigned    wm       0                0         (0/0/0)            0
Do you wish to continue creating a new partition
table based on above table[yes]? yes
Free Hog partition[6]?
Warning: No space available from Free Hog partition.
Continue[no]? yes
Enter size of partition '0' [419441085b, 8703c, 204805.22mb, 200.01gb]:
Enter size of partition '1' [419441085b, 8703c, 204805.22mb, 200.01gb]:
Enter size of partition '3' [419441085b, 8703c, 204805.22mb, 200.01gb]:
Enter size of partition '4' [419441085b, 8703c, 204805.22mb, 200.01gb]:
Warning: no space available for '5' from Free Hog partition
Warning: no space available for '7' from Free Hog partition
Part      Tag    Flag     Cylinders         Size            Blocks
 0 unassigned    wm       0 -  8702      200.01GB    (8703/0/0) 419441085
 1 unassigned    wm    8703 - 17405      200.01GB    (8703/0/0) 419441085
 2 unassigned    wm   17406 - 26108      200.01GB    (8703/0/0) 419441085
 3 unassigned    wm   17406 - 26108      200.01GB    (8703/0/0) 419441085
 4 unassigned    wm   26109 - 34811      200.01GB    (8703/0/0) 419441085
 5 unassigned    wm       0                0         (0/0/0)            0
 6 unassigned    wm       0                0         (0/0/0)            0
 7 unassigned    wm       0                0         (0/0/0)            0
Okay to make this the current partition table[yes]? yes
Enter table name (remember quotes): "extarray"
Ready to label disk, continue? yes
Please answer the prompts as above. you can quit partition util now and go back to format and may save the current partition information in a file for future reference, you can do this by
using save command at format prompt
format> save
Saving new disk and partition definitions
Enter file name["./format.dat"]:
format> quit
Now that partitions are created we need to create file system on them which is pretty straight forward:
#newfs -v /dev/dsk/c2t0d0s0
#newfs -v /dev/dsk/c2t0d0s1
#newfs -v /dev/dsk/c2t0d0s2
#newfs -v /dev/dsk/c2t0d0s3
#newfs -v /dev/dsk/c2t0d0s4

You will see the file system creation messages as we have uses Verbose switch as part of newfs. Only thing remaining now is add this disk information to /etc/vfstab, whcich mean add entries similar to tehse in /etc/vfstab

/dev/dsk/c2t0d0s0       /dev/rdsk/c2t0d0s0 /ora01       ufs     3       yes     -

/dev/dsk/c2t0d0s1       /dev/rdsk/c2t0d0s1 /ora02       ufs     3       yes     -

/dev/dsk/c2t0d0s2       /dev/rdsk/c2t0d0s2 /ora03       ufs     3       yes     -

/dev/dsk/c2t0d0s3       /dev/rdsk/c2t0d0s3 /ora04       ufs     3       yes     -

/dev/dsk/c2t0d0s4       /dev/rdsk/c2t0d0s4 /oraarch     ufs     3       yes     -

save them and do

#mount -a

Newly created partitons should now be mounted and you can verify this by doing

#df -kh

# df -kh | grep ora

/dev/dsk/c2t0d0s4      197G    16G   179G     9%    /oraarch

/dev/dsk/c2t0d0s1      197G    64M   195G     1%    /ora02

/dev/dsk/c2t0d0s0      197G    64M   195G     1%    /ora01

/dev/dsk/c2t0d0s3      197G    64M   195G     1%    /ora04

/dev/dsk/c2t0d0s2      197G    64M   195G     1%    /ora03

And we are done.

The Nice guys at DavMail has written a thin proxy, which you install on your machine and point it to owa url for exchange(most organisations have it).

Setup is very simple:

1) Download DavMail from http://davmail.sourceforge.net/download.html

2) and install it as instructed at http://davmail.sourceforge.net/macosxsetup.html

Then Download Thunderbird from http://www.mozillamessaging.com/thunderbird/

and configure as instructed at http://davmail.sourceforge.net/thunderbirdmailsetup.html, they have instructions for Windows but the setup is more or less same.

For Ical sync follow:http://davmail.sourceforge.net/osxicalsetup.html

If you followed them, then  you are done, and hey you just got rid of bulky Entourage.

you can sync exchange calendar directly from Thunderbird by installing Thunderbird lightning plug-in and following these steps : http://davmail.sourceforge.net/thunderbirdcalendarsetup.html

Cheers

Vipul

2xHP Proliant hp DL380G5 servers with 2 Nics.
OS: RHEL5.4 (Kernel 2.6.18.164.1) x86_64
DRBD: Shared Block device On which Mysql resides.
Pacemaker: Manages the services on cluster. decides whcih node to start services on and monitors services.
Corosync(OpenAIS): monitors the cluster heartbeat.

Clusterlabs(http://www.clusterlabs.org/) have rpms for pacemaker corosync and heartbeat(ocf components for heartbeat, what is ocf read: http://opencf.org/home.html) in there repo(this is the one i used http://www.clusterlabs.org/rpm/epel-5/x86_64/). The drbd kernel module package in clusterlab repo didn’t install on my kernel, so I created mine), other drbd packages from there repo will install happily, the one we need for our setup is drbd-heartbeat, drbd-pacemaker, although I installed all drbd-*.

Step by Step Setup:

Note: This etup assumes that you are running RHEL5.4, x86_64.

Installation:

On both nodes:
1) add clusterlabs repo that is in /etc/yum.repos.d create a repo file and for baseurl use http://www.clusterlabs.org/rpm/epel-5/x86_64/
(You can chose repo depending on you distribution)
2) I installed my locally compiled drbd-utils, drbd-km rpms
rpm -Uvh http://reponame/vk/drbd-utils-8.3.6-1.x86_64.rpm
rpm -Uvh http://reponame/vk/drbd-km-2.6.18_164.6.1.el5xen-8.3.6-12.x86_64.rpm
3) Also install esmtp rpm it’s needed,
rpm -Uvh ftp://fr.rpmfind.net/linux/EPEL/5/x86_64/libesmtp-1.0.4-5.el5.x86_64.rpm
4) yum -y install pacemaker openais drbd-*
5) yum -y install mysql-server
(You can chose to install mysql later)

Configuration:

DRBD:

On both nodes
1) lvcreate –name=drbd –size=1G VolGroup00
(This drbd metadata partition)
2) lvcreate –name=DRBDDATA –size=100G VolGroup01
(This is actual data partition)
3) Zero out the metadata partition
dd if=/dev/zero of=/dev/VolGroup00/drbd
4) create file system on data drive:
mkfs.ext3 /dev/VolGroup01/DRBDDATA
5) mkdir /service
6) edit /etc/drbd.conf, mine looks like

#
# parameters you _need_ to change are the hostname, device, disk,
# meta-disk, address and port in the “on {}” sections.
#
# you ought to know about the protocol, and the various timeouts.
#
# you probably want to set the rate in the syncer sections

#
# NOTE common pitfall:
# rate is given in units of _byte_ not bit
#

#
# increase timeout and maybe ping-int in net{}, if you see
# problems with “connection lost/connection established”
# (or change your setup to reduce network latency; make sure full
# duplex behaves as such; check average roundtrip times while
# network is saturated; and so on …)
#

skip {
As you can see, you can also comment chunks of text
with a ‘skip[optional nonsense]{ skipped text }’ section.
This comes in handy, if you just want to comment out
some ‘resource {…}’ section:
just precede it with ‘skip’.

The basic format of option assignment is
;

It should be obvious from the examples below,
but if you really care to know the details:

:=
valid options in the respective scope
:= |||…
depending on the set of allowed values
for the respective option.
:= [0-9]+, sometimes with an optional suffix of K,M,G
:= (|\”([^\"\\\n]*|\\.)*\”)+
:= [/_.A-Za-z0-9-]+
}

#
# At most ONE global section is allowed.
# It must precede any resource section.
#
global {
# By default we load the module with a minor-count of 32. In case you
# have more devices in your config, the module gets loaded with
# a minor-count that ensures that you have 10 minors spare.
# In case 10 spare minors are too little for you, you can set the
# minor-count exeplicit here. ( Note, in contrast to DRBD-0.7 an
# unused, spare minor has only a very little overhead of allocated
# memory (a single pointer to be exact). )
#
# minor-count 64;

# The user dialog counts and displays the seconds it waited so
# far. You might want to disable this if you have the console
# of your server connected to a serial terminal server with
# limited logging capacity.
# The Dialog will print the count each ‘dialog-refresh’ seconds,
# set it to 0 to disable redrawing completely. [ default = 1 ]
#
# dialog-refresh 5; # 5 seconds

# You might disable one of drbdadm’s sanity check.
# disable-ip-verification;

# Participate in DRBD’s online usage counter at http://usage.drbd.org
# possilbe options: ask, yes, no. Default is ask. In case you do not
# know, set it to ask, and follow the on screen instructions later.
usage-count yes;
}

#
# The common section can have all the sections a resource can have but
# not the host section (started with the “on” keyword).
# The common section must precede all resources.
# All resources inherit the settings from the common section.
# Whereas settings in the resources have precedence over the common
# setting.
#

common {
syncer { rate 1000M; }
net {
cram-hmac-alg “sha1″;
shared-secret “somesecret”;
}
}
resource data1 {
protocol C;
handlers {
pri-on-incon-degr “echo o > /proc/sysrq-trigger ; halt -f”;
pri-lost-after-sb “echo o > /proc/sysrq-trigger ; halt -f”;
local-io-error “echo o > /proc/sysrq-trigger ; halt -f”;
outdate-peer “/usr/lib/heartbeat/drbd-peer-outdater -t 5″;
}
startup {
degr-wfc-timeout 120; # 2 minutes.
}
disk {
on-io-error detach;
}
net {
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}
syncer {
rate 1000M;
al-extents 257;
}
on node1.domain.com {
device /dev/drbd1;
disk /dev/VolGroup01/DRBDDATA;
address 192.168.1.1:7789;
meta-disk /dev/VolGroup00/drbd [0];
}
on node2.domain.com {
device /dev/drbd1;
disk /dev/VolGroup01/DRBDDATA;
address 192.168.1.2:7789;
meta-disk /dev/VolGroup00/drbd [0];
}
}

I connected eth1 interfaces of both servers to each other via a straight through cable and assigned them IPs 192.168.1.1 and 192.168.1.2. This Meant a dedicated sync path and physical connection for DRBD. Please note your drbd.conf should look same on both nodes so best is to edit/create it on one node and then copy it over to other node.

7) modprobe drbd
8) drbdadm create-md data1
drbdadm attach data1
drbadm syncer data1
9) on the node you want to make primary do:
drbdadm — –overwrite-data-of-peer primary data1

for my 100Gb disk it took 13 mins for initial sync. DRBD is up now, you can check this by doing cat /proc/drbd on both nodes and the output should look like:
cat /proc/drbd version: 8.3.6 (api:88/proto:86-91)
GIT-hash: f3606c47cc6fcf6b3f086e425cb34af8b7a81bbf build by root, 2009-12-17 15:47:29

1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r—-
ns:0 nr:245952 dw:245952 dr:0 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0

If it’s primary node you’ll see Primary/Secondary and on secondary it should look like Secondary/Primary , I have paste output from secondary node hence it says Secondary/Primary.

10) on one node do
corosync-keygen
this will generate /etc/corosync/authkey file, copy it to same location on other node.
11) create corosync.conf in /etc/corosync mine looks like

totem {
version: 2
# Disable encryption
secauth: off
# How many threads to use for encryption/decryption
threads: 0
# How long before declaring a token lost (ms)
token: 10000
# How many token retransmits before forming a new configuration
token_retransmits_before_loss_const: 20
# How long to wait for join messages in the membership protocol (ms)
join: 60
# How long to wait for consensus to be achieved before starting a new round of membership configuration (ms)
consensus: 4800
# Turn off the virtual synchrony filter
vsftype: none
# Number of messages that may be sent by one processor on receipt of the token
max_messages: 20
# Limit generated nodeids to 31-bits (positive signed integers)
clear_node_high_bit: yes
# Optionally assign a fixed node id (integer)
#nodeid: 1234
rrp_mode: passive
interface {
# The following values need to be set based on your environment
ringnumber: 0
bindnetaddr: 10.52.72.0
mcastaddr: 226.94.1.5
mcastport: 5406
}
interface {
ringnumber: 1
bindnetaddr: 192.168.1.0
mcastaddr: 226.94.1.6
mcastport: 5407
}

}

logging {
to_stderr: yes
debug: off
timestamp: on
to_file: no
to_syslog: yes
syslog_facility: daemon
}

amf {
mode: disabled
}

service {
# Load the Pacemaker Cluster Resource Manager
ver: 0
name: pacemaker
use_mgmtd: yes
}

aisexec {
user: root
group: root
}

As you can see I have two rings configured for heartbeat.

12) start corosync
/etc/inid.d/corosync start

13) edit you mysql config to point datadir to /service
14) at this point if you execute
crm_mon

you’ll see something like

============
Last updated: Wed Jan 20 20:58:56 2010
Stack: openais
Current DC: amix1800.national.core.bbc.co.uk – partition with quorum
Version: 1.0.7-b1191b11d4b56dcae8f34715d52532561b875cd5
2 Nodes configured, 2 expected votes
0 Resources configured.
============

Online: [ node1.whatever node2.whatever ]

13) configure Cluster Service

#crm configure
this’ll take you to crm configure shell

crm(live)configure#

now type edit and press enter

crm(live)configure# edit

This’ll open config in your default editor and you’ll see a very minimal config, mine after configuration looks like:
you can copy paste this(off course yours’ll be different IP and hostnames)

primitive drbd_mysql ocf:heartbeat:drbd \
params drbd_resource=”data1″ \
op start interval=”0″ timeout=”240s” \
op promote interval=”0″ timeout=”90s” \
op demote interval=”0″ timeout=”90s” \
op notify interval=”0″ timeout=”90s” \
op stop interval=”0″ timeout=”100s” \
op monitor interval=”10s” role=”Master” timeout=”20s” start-delay=”1m” depth=”0″
primitive fs_mysql ocf:heartbeat:Filesystem \
params device=”/dev/drbd1″ directory=”/service/” fstype=”ext3″ \
op start interval=”0″ timeout=”60s” \
op stop interval=”0″ timeout=”60s” \
op notify interval=”0″ timeout=”60s” \
op monitor interval=”20s” timeout=”40s” start-delay=”0″
primitive ip_mysql ocf:heartbeat:IPaddr2 \
params ip=”1.2.3.4″ nic=”eth0″ \
op start interval=”0″ timeout=”90s” \
op stop interval=”0″ timeout=”100s” \
op status interval=”10s” timeout=”20s” start-delay=”0″ \
op monitor interval=”10s” timeout=”20s” start-delay=”0″
primitive mysqld lsb:mysqld
group mysql fs_mysql ip_mysql mysqld
ms ms_drbd_mysql drbd_mysql \
meta master-max=”1″ master-node-max=”1″ clone-max=”2″ clone-node-max=”1″ notify=”true”

colocation mysql_on_drbd inf: mysql ms_drbd_mysql:Master
order mysql_after_drbd inf: ms_drbd_mysql:promote mysql:start

no-quorum-policy=”ignore” \
stonith-enabled=”false”

save and exit

on crm cofnig shell type commit

crm(live)configure# commit

and then exit the shell by typing exit

now if you do crm_mon
you’ll see something like:

============
Last updated: Wed Jan 20 21:07:07 2010
Stack: openais
Current DC: node1.whatever – partition with quorum
Version: 1.0.7-b1191b11d4b56dcae8f34715d52532561b875cd5
2 Nodes configured, 2 expected votes
2 Resources configured.
============

Online: [ node1.whatever node2.whatever ]

Resource Group: mysql
fs_mysql (ocf::heartbeat:Filesystem): Started amix1800.national.core.bbc.co.uk
ip_mysql (ocf::heartbeat:IPaddr2): Started amix1800.national.core.bbc.co.uk
mysqld (lsb:mysqld): Started node1.whatever
Master/Slave Set: ms_drbd_mysql
Masters: [ node1.whatever ]
Slaves: [ node2.whatever ]

You can test you setup by trying a migration of services or bringing down eth0.

If you use IPtables on your systems you’ll need to open relevant ports which are visible in cofigs, also please open udp and tcp both ports.
You could also write your own ocf’s to make your application cluster aware.
Mine is a Master/slave setup, you could create n+1 cluster with these using a cluster file-system(GFS/OCFS etc).
For Further Readings:
1) Linbit has very good documentation on DRBD: http://www.drbd.org/users-guide/
2) Corosync: http://www.corosync.org/doku.php
3) http://www.clusterlabs.org/wiki/Main_Page
4) a master master cluster using same technologies
http://www.creativeanvil.com/blog/2008/how-to-create-an-iscsi-san-using-heartbeat-drbd-and-ocfs2/
6) http://www.openais.org/doku.php

The future looks bright

11:54:34 amix04-07 kernel: printk: 1 messages suppressed.
Sep 29 11:54:34 amix04-07 kernel: ip_conntrack: table full, dropping packet.
Sep 29 11:54:41 amix04-07 kernel: printk: 2 messages suppressed.
Sep 29 11:54:41 amix04-07 kernel: ip_conntrack: table full, dropping packet.
Sep 29 11:54:47 amix04-07 kernel: printk: 3 messages suppressed.
Sep 29 11:54:47 amix04-07 kernel: ip_conntrack: table full, dropping packet.
Sep 29 11:54:48 amix04-07 kernel: ip_conntrack: table full, dropping packet.
Sep 29 11:54:53 amix04-07 kernel: printk: 7 messages suppressed.
Sep 29 11:54:53 amix04-07 kernel: ip_conntrack: table full, dropping packet.
Sep 29 11:54:57 amix04-07 kernel: printk: 12 messages suppressed.
Sep 29 11:54:57 amix04-07 kernel: ip_conntrack: table full, dropping packet.
Sep 29 11:55:03 amix04-07 kernel: printk: 23 messages suppressed.

This happened because the limit of 32000 exceeded for ip_conntrack. I noticed that the both the servers(amix04-07 and amix06-07) crashed after few minutes.

I changed increased the limit to 60000, and looked fine.

this is what I did to change the limit:# This tell you how many sessions arte open right now.
cat /proc/net/ip_conntrack | wc -l

1. This tells you the maximum number of conntrack entries you can have in
total
cat /proc/sys/net/ipv4/ip_conntrack_max

2. This chnages the maximum number of conntrack entries you can have

echo “60000″ > /proc/sys/net/ipv4/ip_conntrack_max

Also remember to add the following line to /etc/sysctl.conf file, to retain the settings after reboot.

net.ipv4.netfilter.ip_conntrack_max=60000

1) Munin (Monitoring tool for, specially good plug ins for Mysql, that’s waht I ahve been told )
2) Cobbler (a tool to manage server builds)
3) Xen 3.4.0 on RHEL5.2
4) Integration with Redhat staellite servers…

yay a lot of new stuff ….. I am excited

Few things i have in mind:

1) Use a standalone db instance for MQ
2) Upgrade our install to latest 5.2 baseline release
3) Upgrade the MySql Jdbc driver to latest 5.1.8 from 5.0.8

I have done above three on dev environment hopefully it helps, a new release of a crucial app is due today, fingers crossed.

I shall post the findings here

As promised I ma updating this with my findings:

1) I have moved database to standalone db server serving just message Active MQ, this has certainly improved performance.
2) have setup 5.2 baseline release, I am not sure if this helped at all, I am hoping it did, specially with stomp fixes However as soon as I upgraded MQ version I started seeing some lovely exceptions whining about persistence adpator , I ahd to disable journaling to overcome this, BAD but only choice.
3) I have upgraded the database driver to 5.1.8, hope that helps somehow .

Now I am waiting for our testers to break it.

I shall update on that soon
cheers
V

To,
Head Office, Woodlands Restaurants,
12, West Close, Wembley Park
Middlessex
HA9 9PJ
Dear Sirs,
I have been a long time visitor of your restaurant in Piccadilly Circus. You Dosa’s are just awesome. Yesterday (Friday 29/05/09) as well I went there in evening with two of my friends in same expectations however Our first order took more then 40 minutes which was just a Delhi thali, onion rava masala dosa, a parantha and a portion of Idlies, We wanted to order some more food so we placed order for one more thali, within 1/2 minutes of that we thought it wouldn’t be good ides to for a thali again so we went upto till to ask if we could change our order, we were rudely denied by a guy claiming that he is the manager. So we politely asked to cancel rest of our order and give us the bill for current one, On this the so called manager went furious saying that we have to pay whether we receive the food or not, He started abusing us in front of the evening crowd for you esteemed restaurant chain.
While we were paying we said that these are appalling standards of customer services, he stated swearing even more and threatening us that he’ll call police. Knowing Our rights I said, We’ll be more then happy to call the police and explain. I mean he was never ready to acknowledge his mistake, you can ask your staff present at that time, they had more manners then there Manager. I mean come on Doesn’t he know that he belongs to heritability industry.
After all this drama he finally said sorry to us, we paid the bill and as we were walking out he again sweard saying “go to hell”, We We reminded him that he is not doing any good to his business by abusing and threatening us.
I was very disappointed with all this show and really felt very embarrassed.

Thanks
Vipul K

NB: I have sent en email to your inquiries email address already, without any response, it seems no one reads them.