Unable to start HADR reason code 7

[gumby]

I'm having trouble getting HADR to work with the sample databases on
two HS20 xSeries blades, Red Hat ES4 up3, DB2 8.2.4, getting the
following error.

SQL1768N Unable to start HADR. Reason code = "7" - The primary
database failed to establish a connection to its standby database
within the HADR timeout interval.

What things should I check besides the remote host and remote service
parameters on the standby database, which seem to be correct. Each of
the servers can see each other via pings etc. I have sucessfully setup
HADR on a single server.

thanks
dub

[Mark A]

"gumby" <da******@gmail .com> wrote in message
news:11******** **************@ i39g2000cwa.goo glegroups.com.. .

I'm having trouble getting HADR to work with the sample databases on
two HS20 xSeries blades, Red Hat ES4 up3, DB2 8.2.4, getting the
following error.

SQL1768N Unable to start HADR. Reason code = "7" - The primary
database failed to establish a connection to its standby database
within the HADR timeout interval.

What things should I check besides the remote host and remote service
parameters on the standby database, which seem to be correct. Each of
the servers can see each other via pings etc. I have sucessfully setup
HADR on a single server.

thanks
dub

I assume you have already started HADR on the standby database first, before
you started HADR on the primary. If that is true, then try logging on the
standby database and activating the standby database

db2 activate database sample

Then retry starting HADR on primary.

[gumby]

Yes, I think the control center runs the following commands anyway. And
if I activate the standby it says it is already activated. Here are the
final commands run.

-- Start HADR on standby database
--
DEACTIVATE DATABASE SAMPLE
START HADR ON DATABASE SAMPLE AS STANDBY
--
-- Start HADR on primary database
--
DEACTIVATE DATABASE SAMPLE
START HADR ON DATABASE SAMPLE AS PRIMARY

Just to clarify, I have sucessfully setup HADR bewteen 2 different
databases on the same server using the control center gui. My problem
is between databases on two different servers. I have tried the manual
command method and the control center, both with the same results.

Using the control center commands

Standby diag file ends with

2006-05-31-16.24.26.101725-240 E476637G362 LEVEL: Event
PID : 27068 TID : 3086558912 PROC : db2hadrs
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrSetHdrState,
probe:10000
CHANGE : HADR state set to S-RemoteCatchupPe nding (was S-LocalCatchup)

2006-05-31-16.24.25.999932-240 I477000G398 LEVEL: Warning
PID : 27057 TID : 3086558912 PROC : db2agent
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
APPHDL : 0-14 APPID: *LOCAL.sample.0 60531202426
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduStartup,
probe:21152
MESSAGE : Info: HADR Startup has completed.

Primary diag files ends with

2006-05-31-16.24.32.714718 +600 E128512G336 LEVEL: Event
PID : 9575 TID : 3085870784 PROC : db2hadrp
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrSetHdrState,
probe:10000
CHANGE : HADR state set to P-Boot (was None)

2006-05-31-16.24.32.719416 +600 I128849G318 LEVEL: Warning
PID : 9575 TID : 3085870784 PROC : db2hadrp
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduP,
probe:20301
MESSAGE : Info: Primary Started.

2006-05-31-16.26.18.769577 +600 I129489G321 LEVEL: Event
PID : 5376 TID : 2947414960 PROC : db2hmon
INSTANCE: sample NODE : 000
FUNCTION: DB2 UDB, Automatic Table Maintenance, db2HmonEvalStat s,
probe:900
STOP : Automatic Runstats: evaluation has finished on database
SAMPLE

2006-05-31-16.26.33.712145 +600 I129811G571 LEVEL: Error
PID : 9575 TID : 3085870784 PROC : db2hadrp
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduP,
probe:20390
MESSAGE : HADR primary did not establish connection with standby within
timeout
and will shut down. BY FORCE option required to start primary
without
standby. Timeout seconds =
DATA #1 : Hexdump, 4 bytes
0x12C13A3C : 7800 0000 x...

2006-05-31-16.26.33.712399 +600 I130383G418 LEVEL: Error
PID : 9575 TID : 3085870784 PROC : db2hadrp
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduP,
probe:20390
RETCODE : ZRC=0x8280001A=-2105540582=HDR_ ZRC_NO_STANDBY
"Comm time-out in unforced HADR primary start, to avoid
split-brain"

2006-05-31-16.26.33.712573 +600 I130802G319 LEVEL: Warning
PID : 9575 TID : 3085870784 PROC : db2hadrp
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduP,
probe:20302
MESSAGE : Info: Primary Finished.

2006-05-31-16.26.33.712704 +600 I131122G422 LEVEL: Error
PID : 9575 TID : 3085870784 PROC : db2hadrp
(SAMPLE) 0
INSTANCE: sample NODE : 000 DB : SAMPLE
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrEduEntry,
probe:21100
RETCODE : ZRC=0x8280001A=-2105540582=HDR_ ZRC_NO_STANDBY
"Comm time-out in unforced HADR primary start, to avoid
split-brain"
Any assistance greatly appreciated

cheers
dub
dub

[Mark A]

"gumby" <da******@gmail .com> wrote in message
news:11******** **************@ u72g2000cwu.goo glegroups.com.. .

Yes, I think the control center runs the following commands anyway. And
if I activate the standby it says it is already activated. Here are the
final commands run.

-- Start HADR on standby database
--
DEACTIVATE DATABASE SAMPLE
START HADR ON DATABASE SAMPLE AS STANDBY
--
-- Start HADR on primary database
--
DEACTIVATE DATABASE SAMPLE
START HADR ON DATABASE SAMPLE AS PRIMARY

Just to clarify, I have sucessfully setup HADR bewteen 2 different
databases on the same server using the control center gui. My problem
is between databases on two different servers. I have tried the manual
command method and the control center, both with the same results.

Any assistance greatly appreciated

cheers
dub

Can you post your db config parms (HADR section only) on both primary and
standby databases?

Also, post output from "db2level" and the OS you are using.

[Steve Pearson (news only)]

From the snippets of diag log shown, it appears that the standby was

not able to establish a socket connection with the primary (primary
listens, standby connects). It seems fairly common that this is not
correctly configured on the first attempt. We've seen issues with
incorrect HADR parameters, DNS problems, failure to properly set up
service names, and inability to correctly map across a NAT.

Double check that your HADR comms parameters mesh up correctly (each
side properly refers to itself in LOCAL params and to the other in
REMOTE params).

HADR_LOCAL_HOST
HADR_LOCAL_SVC
HADR_REMOTE_HOS T
HADR_REMOTE_SVC

Ensure that your service names are registered and/or use IP addresses.
Try using fully-specified network naming (a.b.c.d) for host names if
you haven't already.

HTH.

Regards,
- Steve P.
--
Steve Pearson, IBM DB2 UDB for LUW Development, IBM Software Group
DB2 "Portland" Team, IBM Beaverton Lab, Beaverton, OR, USA

[Phil Sherman]

Your ability to get this working on a single system indicates that you
have the knowledge to do this from the database perspective.

A common cause of problems when going from one system to two systems,
especially with Linux, is the requirement to pass through the firewalls.
Make sure they are configured to allow the HADR ports to pass traffic.

Phil Sherman

gumby wrote:

I'm having trouble getting HADR to work with the sample databases on
two HS20 xSeries blades, Red Hat ES4 up3, DB2 8.2.4, getting the
following error.

SQL1768N Unable to start HADR. Reason code = "7" - The primary
database failed to establish a connection to its standby database
within the HADR timeout interval.

What things should I check besides the remote host and remote service
parameters on the standby database, which seem to be correct. Each of
the servers can see each other via pings etc. I have sucessfully setup
HADR on a single server.

thanks
dub

[Mark A]

"Phil Sherman" <ps******@ameri tech.net> wrote in message
news:R8******** **@newssvr33.ne ws.prodigy.com. ..

Your ability to get this working on a single system indicates that you
have the knowledge to do this from the database perspective.

A common cause of problems when going from one system to two systems,
especially with Linux, is the requirement to pass through the firewalls.
Make sure they are configured to allow the HADR ports to pass traffic.

Phil Sherman

He is using the GUI interface. I was able to configure HADR on a local
Windows box with the GUI, but not with remote Linux boxes. Using command
line configuration scripts on remote Linux boxes worked fine.

[gumby]

I'm Running Red Hat ES4 up3

[sample@tank ~]$ uname -r
2.6.9-34.ELsmp

[sample@tank ~]$ db2level
DB21085I Instance "sample" uses "32" bits and DB2 code release
"SQL08024" with
level identifier "03050106".
Informational tokens are "DB2 v8.1.2.104", "s060120", "MI00152", and
FixPak
"11".
Product is installed at "/opt/IBM/db2/V8.1".

STANDBY - tank
HADR database role = STANDARD
HADR local host name (HADR_LOCAL_HOS T) = tank
HADR local service name (HADR_LOCAL_SVC ) = DB2_HADR_2
HADR remote host name (HADR_REMOTE_HO ST) = dozer
HADR remote service name (HADR_REMOTE_SV C) = DB2_HADR_1
HADR instance name of remote server (HADR_REMOTE_IN ST) = sample
HADR timeout value (HADR_TIMEOUT) = 120
HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC

PRIMARY - dozer
HADR database role = STANDARD
HADR local host name (HADR_LOCAL_HOS T) = dozer
HADR local service name (HADR_LOCAL_SVC ) = DB2_HADR_1
HADR remote host name (HADR_REMOTE_HO ST) = tank
HADR remote service name (HADR_REMOTE_SV C) = DB2_HADR_2
HADR instance name of remote server (HADR_REMOTE_IN ST) = sample
HADR timeout value (HADR_TIMEOUT) = 120
HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC

/etc/services
# Local services
DB2_sample 60000/tcp
DB2_sample_1 60001/tcp
DB2_sample_2 60002/tcp
DB2_sample_END 60003/tcp
DB2_HADR_1 55001/tcp
DB2_HADR_2 55002/tcp

Currently doing some more tests with and without the GUI on the linux
boxes.

[gumby]

Are there any requirements for the servers to be cataloged. I mean on
the primary, should there be a catalog/node entry (no sure of the
correct terms) to the standby. And likewise should there be an entry on
the standby pointing to the primary.

Should they be described by running the command db2 list node
directory.?

[Mark A]

"gumby" <da******@gmail .com> wrote in message
news:11******** *************@y 43g2000cwc.goog legroups.com...

I'm Running Red Hat ES4 up3

[sample@tank ~]$ uname -r
2.6.9-34.ELsmp

[sample@tank ~]$ db2level
DB21085I Instance "sample" uses "32" bits and DB2 code release
"SQL08024" with
level identifier "03050106".
Informational tokens are "DB2 v8.1.2.104", "s060120", "MI00152", and
FixPak
"11".
Product is installed at "/opt/IBM/db2/V8.1".

STANDBY - tank
HADR database role = STANDARD
HADR local host name (HADR_LOCAL_HOS T) = tank
HADR local service name (HADR_LOCAL_SVC ) = DB2_HADR_2
HADR remote host name (HADR_REMOTE_HO ST) = dozer
HADR remote service name (HADR_REMOTE_SV C) = DB2_HADR_1
HADR instance name of remote server (HADR_REMOTE_IN ST) = sample
HADR timeout value (HADR_TIMEOUT) = 120
HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC

PRIMARY - dozer
HADR database role = STANDARD
HADR local host name (HADR_LOCAL_HOS T) = dozer
HADR local service name (HADR_LOCAL_SVC ) = DB2_HADR_1
HADR remote host name (HADR_REMOTE_HO ST) = tank
HADR remote service name (HADR_REMOTE_SV C) = DB2_HADR_2
HADR instance name of remote server (HADR_REMOTE_IN ST) = sample
HADR timeout value (HADR_TIMEOUT) = 120
HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC

/etc/services
# Local services
DB2_sample 60000/tcp
DB2_sample_1 60001/tcp
DB2_sample_2 60002/tcp
DB2_sample_END 60003/tcp
DB2_HADR_1 55001/tcp
DB2_HADR_2 55002/tcp

Currently doing some more tests with and without the GUI on the linux
boxes.

This may not help, but I would use the port number in your db config, and
not the service name (but leave the service names in the /etc/services).

I assume the database names are the same on primary and standby (not
specified in your post above).

When the HADR database role is STANDARD, then that means that HADR has not
been started. So manually "start HADR on db xxxxxxx as standby" (on tank),
and then (if successful) "start HADR on db xxxxxxx as primary" (on dozer).
You must start HADR on the standby first.

If the above does not work, then you should check the ports (at the OS
level) to make sure no one else is using 55001 and 55002. The recommended
HADR ports start with 18819 (although I have no reason why, and don't know
if this matters).

A useful monitoring tool of the current HADR status without the GUI is to
take a database snapshot (refer to HADR section):
db2 get snapshot for database on xxxxxxxx

As I said previously, I was not able to get the GUI to work for HADR on
Linux, but there are very few commands needed to get it working, so it is
easy to script from the command line.