Symptoms

restarting after kill -9 of CSC instance does not succeed. The server never starts, the WAR files are not deployed properly, socket 8080 is open but nobody answers (not even with wget on localhost).

The exception:

"main" prio=10 tid=0x0000000051a39000 nid=0x30fb in Object.wait() [0x00002b5df9d6c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007e36ae9d0> (a atg.service.lockmanager.ClientLockEntry)
    at atg.service.lockmanager.ClientLockEntry.acquireWriteLock(ClientLockEntry.java:399)
    - locked <0x00000007e36ae9d0> (a atg.service.lockmanager.ClientLockEntry)
    at atg.service.lockmanager.ClientLockManager.acquireWriteLock(ClientLockManager.java:1137)
    at atg.repository.search.indexing.ConfigStatePersister.acquireLock(ConfigStatePersister.java:1844)
    at atg.repository.search.indexing.ConfigStatePersister$IndexingReadLockedTransactionAction.doWithTransaction(ConfigStatePersister.java:2083)
    at atg.repository.search.indexing.ConfigStatePersister.releaseAllConfigurations(ConfigStatePersister.java:1545)
    at atg.repository.search.indexing.ConfigStatePersister.performAction(ConfigStatePersister.java:1775)
    at atg.nucleus.Nucleus.performPostStartupActions(Nucleus.java:4336)
    at atg.nucleus.Nucleus.performPostStartupActions(Nucleus.java:4323)
    at atg.nucleus.Nucleus.<init>(Nucleus.java:954)
    at atg.nucleus.Nucleus.<init>(Nucleus.java:717)
    at atg.nucleus.Nucleus.startNucleusCheckLicense(Nucleus.java:4144)
    at atg.nucleus.servlet.NucleusServlet.initBigEarNucleus(NucleusServlet.java:1234)
    at atg.nucleus.servlet.NucleusServlet.init(NucleusServlet.java:443)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1048)
    - locked <0x0000000781734f48> (a org.apache.catalina.core.StandardWrapper)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:950)
    - locked <0x0000000781734f48> (a org.apache.catalina.core.StandardWrapper)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4122)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4417)
    - locked <0x0000000781733e18> (a org.apache.catalina.core.StandardContext)
    at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeployInternal(TomcatDeployment.java:321)
    at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeploy(TomcatDeployment.java:145)
    at org.jboss.web.deployers.AbstractWarDeployment.start(AbstractWarDeployment.java:461)
    - locked <0x0000000780878058> (a org.jboss.web.tomcat.service.deployers.TomcatDeployment)
    at org.jboss.web.deployers.WebModule.startModule(WebModule.java:118)
    - locked <0x0000000780878038> (a org.jboss.web.deployers.WebModule)
    at org.jboss.web.deployers.WebModule.start(WebModule.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
    at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)
    at $Proxy38.start(Unknown Source)
    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
    at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
    at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
    at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
    at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
    at org.jboss.system.ServiceController.start(ServiceController.java:460)
    at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
    at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
    at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)
    at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1225)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)
    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
    at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)
    at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)
    at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
    at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
    at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
    at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)
    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)
    at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
    at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)
    at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
    at org.jboss.Main.boot(Main.java:221)
    at org.jboss.Main$1.run(Main.java:556)
    at java.lang.Thread.run(Thread.java:662)

What is going on:

In the run of deployment, JBoss is starting context, hands over control to NucleusServlet init. Nucleus is trying to get write access lock and hangs there forever, while WAR deployment and all subsequent deployment never completes.

Really ? Is this the best you can do ?

Why would not JBOSS time out the loadServlet action, catch exception and continue deploying the other WAR file ?

From Nucleus point of view, depending on presence (and even worse WAITING for it) of external service to start up is not very bright idea. It would be much more helpful throw exception indicating there is not a LockServer available and abort the deployment.

Just hanging there is plain stupid.

To add to the confusion, some of the service from the WAR file that is deployed are running and spitting exceptions to log file:

21:01:55,410 ERROR [/atg/ticketing/scheduled/UnsuspendTicketPeriodicService] Error while processing item ticket:105397
atg.ticketing.TicketingException: Cannot unsuspend ticket because it is not pending.
    at atg.ticketing.TicketingManager$7.doInTransaction(TicketingManager.java:2164)
    at atg.dtm.TransactionUtil$TypedTransactionAction.doWithTransaction(TransactionUtil.java:73)
    at atg.ticketing.TicketingManager.unsuspendTicket(TicketingManager.java:2155)
    at atg.ticketing.scheduled.UnsuspendTicketPeriodicService.processDueItem(UnsuspendTicketPeriodicService.java:116)
    at atg.ticketing.scheduled.TicketingPeriodicService$1.doInTransaction(TicketingPeriodicService.java:193)
    at atg.dtm.TransactionUtil$TypedTransactionAction.doWithTransaction(TransactionUtil.java:73)
    at atg.ticketing.scheduled.TicketingPeriodicService.performScheduledTask(TicketingPeriodicService.java:191)
    at atg.ticketing.scheduled.TicketingPeriodicService.performScheduledTask(TicketingPeriodicService.java:165)
    at atg.service.scheduler.ScheduledJob.runJobs(ScheduledJob.java:441)
    at atg.service.scheduler.Scheduler$2handler.run(Scheduler.java:760)

Again - why should the scheduled services be started BEFORE we know the EAR/WAR has actually successfully deployed ? These are not the services required for the boot !!!