Home » RDBMS Server » Networking and Gateways » Help !!! - listener state:blocked (Oracle RAC 10.1.0.3)
icon8.gif  Help !!! - listener state:blocked [message #342629] Sat, 23 August 2008 07:01 Go to next message
jbourke
Messages: 6
Registered: August 2008
Junior Member
Folks,

My listener is blocked. And it does not seem to be for the usual reasons which I have seen on the Web.

It is blocked directly after startup, when there are no user connections. So it is not a PROCESSES issue.

LSNRCTL> services LISTENER_POPB
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=212.165.120.13)(PORT=1521)))
Services Summary...
Service "hhme" has 2 instance(s).
Instance "hhme1", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPA)(PORT=1521))
Instance "hhme2", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:blocked
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPB)(PORT=1521))
Service "hhmeXDB" has 2 instance(s).
Instance "hhme1", status READY, has 1 handler(s) for this service...
Handler(s):
"D000" established:0 refused:0 current:0 max:972 state:ready
DISPATCHER <machine: PoPA, pid: 5260>
(ADDRESS=(PROTOCOL=tcp)(HOST=PoPA.CleanLANA.bgan.xantic.net)(PORT=32797))
Instance "hhme2", status READY, has 1 handler(s) for this service...
Handler(s):
"D000" established:0 refused:0 current:0 max:1022 state:ready
DISPATCHER <machine: PoPB, pid: 6543>
(ADDRESS=(PROTOCOL=tcp)(HOST=POPB)(PORT=32855))
The command completed successfully

For a user connection, in the listener log, I get

[oracle@PoPB log]$ tail -f listener_popb.log
23-AUG-2008 11:24:14 * (CONNECT_DATA=(SERVICE_NAME=HHME)(INSTANCE_NAME=HHME2)(CID=(PROGRAM=C:\Program Files\Quest Software\SQL Navigator 5.5\SQLNav5.exe)(HOST=JOHN_LAPTOP)(USER=John?Bourke))) * (ADDRESS=(PROTOCOL=tcp)(HOST=84.9.3.187)(PORT=2659)) * establish * HHME * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack

Which corresponds to the following on the user client

ORA-12516: TNS:listener could not find available handler with matching protocol stack

I have tried cold and warm starts, with the same results.
The A node in the cluster does not have this problem.

On startup, there are no errors in the alert log or listener log.

TNSPINGs work or and crs_stat -t shows everything online, and VIP is in place in the ifconfig.

Can anyone suggest somewhere else to look ?

Thanks

john

Report message to a moderator

Re: Help !!! - listener state:blocked [message #342630 is a reply to message #342629] Sat, 23 August 2008 07:09 Go to previous messageGo to next message
Mohammad Taj
Messages: 2412
Registered: September 2006
Location: Dubai, UAE
Senior Member
Yes, if you metalink account then check.

Report message to a moderator

Re: Help !!! - listener state:blocked [message #342642 is a reply to message #342629] Sat, 23 August 2008 09:03 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
 
12516, 00000, "TNS:listener could not find available handler with matching protocol stack"
// *Cause: None of the known and available service handlers for the given 
// SERVICE_NAME support the client's protocol stack: transport, session, 
// and presentation protocols.
// *Action: Check to make sure that the service handlers (e.g. dispatchers) 
// for the given SERVICE_NAME are registered with the listener, are accepting 
// connections, and that they are properly configured to support the desired 
// protocols.


GOOGLE is your friend but only when you actually use it.

Report message to a moderator

Re: Help !!! - listener state:blocked [message #342656 is a reply to message #342629] Sat, 23 August 2008 14:31 Go to previous messageGo to next message
jbourke
Messages: 6
Registered: August 2008
Junior Member
Hi,

Note: I do not have metalink access at the moment.

I have checked the listener log and service registrations and service updates are happening

23-AUG-2008 11:04:10 * service_register * hhme1 * 0
23-AUG-2008 11:04:49 * service_register * hhme2 * 0

23-AUG-2008 19:08:57 * service_update * hhme1 * 0
23-AUG-2008 19:09:05 * service_update * hhme2 * 0

I have checked the oracle documentation for configuration and debugging of the listener

OracleŽ Database Net Services Administrator's Guide
10g Release 1 (10.1)

There is nothing in this document which I have configured incorrectly, and there is no test or debugging information which I have not tried.

In order to do a gap analysis, I have a working node to compare with, and also I have an identically configured cluster to compare.

The "blocked" server has been running for years without problems. I have searched all the configurations to see if anything was changed recently, but I cannot find any changes. I am also cross checking with the working server configs and cannot find any differences or changes.

LSNRCTL services working node shows me

Instance "hhme1", status READY, has 2 handler(s) for this service...
Handler(s):
"DEDICATED" established:5655 refused:0 state:ready
LOCAL SERVER
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPA)(PORT=1521))

LSNRCTL services blocked node shows me

Instance "hhme2", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:blocked
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPB)(PORT=1521))

On the blocked node I am missing

"DEDICATED" established:5655 refused:0 state:ready
LOCAL SERVER

On the identical working cluster, both nodes have a LOCAL SERVER entry in the LSNRCTL services output.

So if registration/updates are ok, should I be chasing this difference ??

Thanks

john

Report message to a moderator

Re: Help !!! - listener state:blocked [message #342657 is a reply to message #342629] Sat, 23 August 2008 14:42 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
Have you run out of available process slots?

Are you using Connection Manager?

Are running in an environment where Java clients are making frequent & very short connections into the DB?

[Updated on: Sat, 23 August 2008 14:47] by Moderator

Report message to a moderator

Re: Help !!! - listener state:blocked [message #342666 is a reply to message #342657] Sat, 23 August 2008 17:02 Go to previous messageGo to next message
jbourke
Messages: 6
Registered: August 2008
Junior Member
Hi,

As the listener is blocked on database start, we have no users logged in.

SQL> select ADDR, PID, SPID, Username from v$process;

ADDR PID SPID USERNAME
-------- ---------- ------------ ---------------
974CC1C8 1
974CC6C4 2 6517 oracle
974CCBC0 3 6519 oracle
974CD0BC 4 6521 oracle
974CD5B8 5 6523 oracle
974CDAB4 6 6525 oracle
974CDFB0 7 6527 oracle
974CE4AC 8 6529 oracle
974CE9A8 9 6531 oracle
974CEEA4 10 6533 oracle
974CF3A0 11 6535 oracle

ADDR PID SPID USERNAME
-------- ---------- ------------ ---------------
974CF89C 12 6537 oracle
974CFD98 13 6539 oracle
974D0294 14 6541 oracle
974D0790 15 6543 oracle
974D0C8C 16 6545 oracle
974D1188 17 6547 oracle
974D1684 18 16320 oracle
974D1B80 19 12857 oracle
974D207C 20 6588 oracle
974D2578 21 6590 oracle
974D2A74 22 6592 oracle

ADDR PID SPID USERNAME
-------- ---------- ------------ ---------------
974D2F70 23 6605 oracle
974D346C 24 6607 oracle
974D3968 25 6609 oracle
974D3E64 26 30700 oracle
974D4360 27 13222 oracle

27 rows selected.

SQL> select * from v$resource_limit where resource_name in
2 ('processes','sessions');

RESOURCE_NAME CURRENT MAX INITIAL
processes 27 30 150
sessions 33 36 170

Not sure if we are using Connection Manager. I do not think so

[oracle@PoPB oracle]$ locate cman.ora

finds nothing.

We connect using JDBC or using quest SQL Navigator. Load Balancing and TAF are not enabled.

We do have a Java app which could be doing a lot of frequent & very short connections, but as the listener is not accepting connections, we have it offline. I guess you are asking this in case there is a connection overload which is hard to spot because connections are too short ?


john

Report message to a moderator

Re: Help !!! - listener state:blocked [message #342667 is a reply to message #342629] Sat, 23 August 2008 17:24 Go to previous messageGo to next message
jbourke
Messages: 6
Registered: August 2008
Junior Member
One more thing.

Our problems started with a database crash. See alert log

Wed Aug 20 12:30:26 2008
Errors in file /u01/app/oracle/admin/hhme/bdump/hhme2_diag_29210.trc:
ORA-07445: exception encountered: core dump [kccdmp_block()+18] [SIGSEGV] [Address not mapped to object] [0x2A3FF584] [] []
Wed Aug 20 12:30:32 2008
Errors in file /u01/app/oracle/admin/hhme/bdump/hhme2_lgwr_29224.trc:
ORA-00600: internal error code, arguments: [kccchb_3], [9311831], [9311834], [9311831], [], [], [], []
Wed Aug 20 12:30:32 2008
Errors in file /u01/app/oracle/admin/hhme/bdump/hhme2_lgwr_29224.trc:
ORA-00600: internal error code, arguments: [kccchb_3], [9311831], [9311834], [9311831], [], [], [], []
Wed Aug 20 12:30:32 2008
LGWR: terminating instance due to error 470
Instance terminated by LGWR, pid = 29224

and listener

20-AUG-2008 12:21:05 * (CONNECT_DATA=(SERVICE_NAME=HHME)(CID=(PROGRAM=C:\Program Files\Quest Software\SQL Navigator 4\SQLNav5.exe
)(HOST=MII-NODE5)(USER=admin))) * (ADDRESS=(PROTOCOL=tcp)(HOST=210.211.249.212)(PORT=3035)) * establish * HHME * 0
20-AUG-2008 12:21:11 * (CONNECT_DATA=(SERVICE_NAME=HHME)(CID=(PROGRAM=C:\Program Files\Quest Software\SQL Navigator 4\SQLNav5.exe
)(HOST=MII-NODE5)(USER=admin))) * (ADDRESS=(PROTOCOL=tcp)(HOST=210.211.249.212)(PORT=3036)) * establish * HHME * 0
20-AUG-2008 12:27:58 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=PoPB)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER_POPB
)(VERSION=168821504)) * status * 0
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=PoPB.virtual.bgan.xantic.net)(PORT=2100))(Presentation=FTP)(Ses
sion=RAW))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=PoPB.virtual.bgan.xantic.net)(PORT=8080))(Presentation=HTTP)(Se
ssion=RAW))
20-AUG-2008 12:30:33 * service_died * hhme2 * 12537
20-AUG-2008 12:30:33 * service_died * hhme2 * 12537
20-AUG-2008 12:30:33 * (CONNECT_DATA=(SID=HHME2)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=212.165.12
0.217)(PORT=30777)) * establish * HHME2 * 12505
TNS-12505: TNS:listener does not currently know of SID given in connect descriptor
20-AUG-2008 12:30:33 * (CONNECT_DATA=(SID=HHME2)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=212.165.12
0.217)(PORT=30778)) * establish * HHME2 * 12505
TNS-12505: TNS:listener does not currently know of SID given in connect descriptor
20-AUG-2008 12:30:33 * (CONNECT_DATA=(SID=HHME2)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=212.165.12
0.217)(PORT=30779)) * establish * HHME2 * 12505

But now the RAC processes and database start OK, no errors. Just this blocked listener.


john

Report message to a moderator

Re: Help !!! - listener state:blocked [message #342668 is a reply to message #342629] Sat, 23 August 2008 17:59 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
http://www.orafaq.com/forum/t/88153/0/
Please read & FOLLOW the Posting Guidelines as stated in URL above

When was the last time things were OK?
What has changed since then?

I suspect/see multiple different errors/problems.
I suspect somebody had a problem & started making changes which made a bad situation worse.
I suspect that there is more to this sad saga, than what has been posted here so far.

Other than that Mrs. Lincoln, how was the theater?

>LGWR: terminating instance due to error 470

Results 1 - 10 of about 202 for LGWR: terminating instance due to error

I am sorry to see that GOOGLE is broken for you.
Please wait for repairs to be completed.
Additiional information will be made available after repairs are done.

[Updated on: Sat, 23 August 2008 18:18] by Moderator

Report message to a moderator

Re: Help !!! - listener state:blocked [message #344667 is a reply to message #342668] Sun, 31 August 2008 18:29 Go to previous message
jbourke
Messages: 6
Registered: August 2008
Junior Member
Hi,

Found it !!!!

The problem was that someone had edited the host file.

Usually the host file has a address to name mapping for the hostname

1.2.3.4 POPB

The change was that someone had added an extra instance of the hostname on the loopback line

127.0.0.1 POPB localhost
1.2.3.4 POPB

Removing the hostname on the loopback line fixed the problem.

I'm not sure I'll ever know what specific oracle behavior results in this action causing status:blocked.

Thanks to everyone for their assistance.


john

Report message to a moderator

Previous Topic: SRVCTL STATUS: Listener is not running [Merged]
Next Topic: Increasing number of sessions in TNS LISTENER
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Fri Apr 19 01:13:20 CDT 2024
.:: Forum Home :: Blogger Home :: Wiki Home :: Contact :: Privacy ::.