ColdFusion J2EE, serviceFactory goes away

[TomChiv]

This is really winding me up, been fighting since Sunday.
The symptom is that the site is fine for a few hours, and then all pages
return a 500 server error.

This is an MX 6.1 with Updater on Weblogic 7 (latest patch set) with JRocket
1.4.2_05 JVM on RedHat AS 2.1

Google isn't being very helpful, and cf-talk is silent on the matter, and
because CF *technically* installs and
runs (for a bit) before crashing, MM tech. support won't talk to me either -
correct ?

The failure mode is that somewhere the ServiceFactory goes gaga ('caused by'
in first log entry below), and then all subsequent requests fail with the
same thing as in the 2nd log entry below.

<12-May-2005 11:38:27 BST> <Error> <HTTP> <101017>
<[ServletContext(id=257353425,name=ceo_platform,cont ext-path=)] Root cause of
ServletException
java.lang.ExceptionInInitializerError
at

coldfusion.filter.RequestMonitorFilter.isFilterEna bled()Z(RequestMonitorFilter.j
ava:???)
at

coldfusion.CfmServlet.getFilterChain(Ljavax.servle t.ServletRequest;)Lcoldfusion.
filter.FusionFilter;
(CfmServlet.java:59)
at

coldfusion.CfmServlet.service(Ljavax.servlet.Servl etRequest;Ljavax.servlet.Servl
etResponse;)V(CfmServlet.java:101)
at

coldfusion.bootstrap.BootstrapServlet.service(Ljav ax.servlet.ServletRequest;Ljav
ax.servlet.ServletResponse;)V(BootstrapServlet.jav a:89)
at

weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run()Ljava.lan
g.Object;
(ServletStubImpl.java:1111)
at

weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(Ljavax.servlet.ServletRe
quest;Ljavax.servlet.ServletResponse;Lweblogic.ser vlet.internal.FilterChainImpl;
)V(ServletStubImpl.java:437)
at

weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(Ljavax.servlet.ServletRe
quest;Ljavax.servlet.ServletResponse;)V(ServletStu bImpl.java:319)
at

weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run()Ljav
a.lang.Object;
(WebAppServletContext.java:5699)
at

weblogic.security.service.SecurityServiceManager.r unAs(Lweblogic.security.acl.in
ternal.AuthenticatedSubject;Lweblogic.security.acl .internal.AuthenticatedSubject
;Ljava.security.PrivilegedAction;)Ljava.lang.Objec t;
(SecurityServiceManager.java:690)
at

weblogic.servlet.internal.WebAppServletContext.inv okeServlet(Lweblogic.servlet.i
nternal.ServletRequestImpl;Lweblogic.servlet.inter nal.ServletResponseImpl;)V(Web
AppServletContext.java:3216)
at

weblogic.servlet.internal.ServletRequestImpl.execu te(Lweblogic.kernel.ExecuteThr
ead;)V(ServletRequestImpl.java:2591)
at

weblogic.kernel.ExecuteThread.execute(Lweblogic.ke rnel.ExecuteRequest;)V(Optimiz
ed
Method)
at weblogic.kernel.ExecuteThread.run()V(ExecuteThread .java:223)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thre ad;)V(Unknown
Source)
Caused by: coldfusion.server.ServiceFactory$ServiceNotAvailab leException: The
Runtime service is not available.
at

coldfusion.server.ServiceFactory.getRuntimeService ()Lcoldfusion.server.RuntimeSe
rvice;
(ServiceFactory.java:107)
at

coldfusion.filter.RequestMonitorFilter.<clinit>()V (RequestMonitorFilter.java:12)
at jrockit.vm.RNI.getRunnableCode(I)I(Unknown Source)
at jrockit.vm.RNI.trampoline()V(Unknown Source)
... 14 more

>

<12-May-2005 11:38:28 BST> <Error> <HTTP> <101017>
<[ServletContext(id=257353425,name=ceo_platform,cont ext-path=)] Root cause of
ServletException
java.lang.NoClassDefFoundError: coldfusion/filter/RequestMonitorFilter
at

coldfusion.filter.RequestMonitorFilter.isFilterEna bled()Z(RequestMonitorFilter.j
ava:???)
at

coldfusion.CfmServlet.getFilterChain(Ljavax.servle t.ServletRequest;)Lcoldfusion.
filter.FusionFilter;
(CfmServlet.java:59)
at

coldfusion.CfmServlet.service(Ljavax.servlet.Servl etRequest;Ljavax.servlet.Servl
etResponse;)V(CfmServlet.java:101)
at

coldfusion.bootstrap.BootstrapServlet.service(Ljav ax.servlet.ServletRequest;Ljav
ax.servlet.ServletResponse;)V(BootstrapServlet.jav a:89)
at

weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run()Ljava.lan
g.Object;
(ServletStubImpl.java:1111)
at

weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(Ljavax.servlet.ServletRe
quest;Ljavax.servlet.ServletResponse;Lweblogic.ser vlet.internal.FilterChainImpl;
)V(ServletStubImpl.java:437)
at

weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(Ljavax.servlet.ServletRe
quest;Ljavax.servlet.ServletResponse;)V(ServletStu bImpl.java:319)
at

weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run()Ljav
a.lang.Object;
(WebAppServletContext.java:5699)
at

weblogic.security.service.SecurityServiceManager.r unAs(Lweblogic.security.acl.in
ternal.AuthenticatedSubject;Lweblogic.security.acl .internal.AuthenticatedSubject
;Ljava.security.PrivilegedAction;)Ljava.lang.Objec t;
(SecurityServiceManager.java:690)
at

weblogic.servlet.internal.WebAppServletContext.inv okeServlet(Lweblogic.servlet.i
nternal.ServletRequestImpl;Lweblogic.servlet.inter nal.ServletResponseImpl;)V(Web
AppServletContext.java:3216)
at

weblogic.servlet.internal.ServletRequestImpl.execu te(Lweblogic.kernel.ExecuteThr
ead;)V(ServletRequestImpl.java:2591)
at

weblogic.kernel.ExecuteThread.execute(Lweblogic.ke rnel.ExecuteRequest;)V(Optimiz
ed
Method)
at weblogic.kernel.ExecuteThread.run()V(ExecuteThread .java:223)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thre ad;)V(Unknown
Source)

[TomChiv]

The only slightly odd thing is that immediately before the initial
ExceptionInInitializerError we have:
####<13-May-2005 06:49:01 BST> <Info> <HTTP> <GBB_AppServer> <ExecuteThread:
'4' for queue: 'default'> <kernel identity> <> <101047> <[ServletContext
(id=157796498,name=ceo_platform,context-path=)] CfmServlet: init>

Shouldn'[t that already be init'ed ? What could make it try to re-init ?

[TomChiv]

Odder and odder, it appears we are consistently getting the below, like
WebLogic is trying to restart ColdFusion but making a total hash of it?

<13-May-2005 16:01:57 BST> <Info> <HTTP> <101047>
<[ServletContext(id=158529739,name=ceo_platform,cont ext-path=)] /*: destroy>
<13-May-2005 16:01:57 BST> <Info> <HTTP> <101047>
<[ServletContext(id=158529739,name=ceo_platform,cont ext-path=)] /*: init>
<13-May-2005 16:01:57 BST> <Info> <HTTP> <101047>
<[ServletContext(id=158529739,name=ceo_platform,cont ext-path=)] /*: Using
standard I/O>
<13-May-2005 16:02:00 BST> <Info> <HTTP> <101047>
<[ServletContext(id=158529739,name=ceo_platform,cont ext-path=)] MapImageHelper:
destroy>
<13-May-2005 16:02:00 BST> <Info> <HTTP> <101047>
<[ServletContext(id=158529739,name=ceo_platform,cont ext-path=)] MapImageHelper:
init>
<13-May-2005 16:02:03 BST> <Info> <HTTP> <101047>
<[ServletContext(id=158529739,name=ceo_platform,cont ext-path=)] CfmServlet:
init>

before our:

<13-May-2005 16:02:03 BST> <Error> <HTTP> <101017>
<[ServletContext(id=158529739,name=ceo_platform,cont ext-path=)] Root cause of
ServletException
java.lang.ExceptionInInitializerError

[TomChiv]

*Anyway*, after fighting into the wee hours all week, we think we have (touch
wood) cracked the F**ker.
Weblogic has a ServletReload parmeter, which says how ofton to check if a
servlet 'has changed' (whatever that means) and reloads it if so. This defaults
to '1' (second), and had been set like that for months.
It has been set to '-1' (don't check ever) for nearly 3 days now, and no
ExceptionInInitializerError followed by 500s.
Rar.
OK, so it fell over this morning with a stack overflow, but I feel that was
our own fault - we need to roll back in the CFMX updater, weblogic SP's etc.
etc.

:the annoyed