Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jetty QueuedThreadPool RejectedExecutionException #20

Closed
dcrankshaw opened this issue Nov 5, 2014 · 2 comments
Closed

Jetty QueuedThreadPool RejectedExecutionException #20

dcrankshaw opened this issue Nov 5, 2014 · 2 comments
Assignees
@dcrankshaw

Comments

@dcrankshaw
Copy link
Contributor

For some reason we are getting RejectedExecutionExceptions thrown in Jetty's QueuedThreadPool.

I'm not sure exactly what's going on, but we only see it when the system is at high load. I'm assuming we've reached the limit of some resource, but I'm still tracking down which resource it is. I suspect it might have something to do with reaching the queue limit or something.

Here is a representative stack trace:

WARN  [2014-11-03 21:38:47,436] edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool: dw{STARTED,8<=124<=1024,i=0,q=1024} rejected AC.ExReadCB@2c056c65
WARN  [2014-11-03 21:38:47,441] edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool: dw{STARTED,8<=124<=1024,i=0,q=1024} rejected AC.ExReadCB@186d0670
WARN  [2014-11-03 21:38:47,675] edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool: dw{STARTED,8<=140<=1024,i=0,q=1024} rejected AC.ExReadCB@d240d9d
WARN  [2014-11-03 21:38:47,996] edu.berkeley.veloxms.jetty.io.SelectorManager: Could not process key for channel java.nio.channels.SocketChannel[connected local=/10.143.219.252:8080 remote=/10.147.1.83:34687]
! java.util.concurrent.RejectedExecutionException: AC.ExReadCB@407e9e8e
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:357) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:407) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.FillInterest.fillable(FillInterest.java:78) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:506) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:463) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
WARN  [2014-11-03 21:38:48,003] edu.berkeley.veloxms.jetty.io.SelectorManager: Could not process key for channel java.nio.channels.SocketChannel[connected local=/10.143.219.252:8080 remote=/10.143.223.222:60187]
! java.util.concurrent.RejectedExecutionException: AC.ExReadCB@d240d9d
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:357) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:407) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.FillInterest.fillable(FillInterest.java:78) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:506) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:463) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
WARN  [2014-11-03 21:38:48,011] edu.berkeley.veloxms.jetty.io.SelectorManager: Could not process key for channel java.nio.channels.SocketChannel[connected local=/10.143.219.252:8080 remote=/10.143.223.222:58371]
! java.util.concurrent.RejectedExecutionException: AC.ExReadCB@186d0670
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:357) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:407) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.FillInterest.fillable(FillInterest.java:78) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109) ~[veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:506) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:463) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at edu.berkeley.veloxms.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [veloxms-core-0.0.1-SNAPSHOT.jar:na]
! at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

Here is the line in Jetty code where the warning logging message is catching the exception and logging. And this the the link to the QueuedThreadPool source code.
@dcrankshaw dcrankshaw changed the title Jetty QueuedThreadPool RejectedExecutionException Jetty QueuedThreadPool RejectedExecutionException Nov 5, 2014
@dcrankshaw dcrankshaw changed the title Jetty QueuedThreadPool RejectedExecutionException Jetty QueuedThreadPool RejectedExecutionException Nov 5, 2014
@dcrankshaw dcrankshaw self-assigned this Nov 5, 2014
@dcrankshaw
Copy link
Contributor Author

I'm still not quite sure why this is happening, but it's definitely the result of exceeding some resource limit. For now I have been avoiding it by throttling the requests on the client side, but it would be good to figure this out. When I see this error, the requests that can't be processed are silently dropped on the floor, which is definitely not the right failure mode.

@dcrankshaw dcrankshaw mentioned this issue Mar 30, 2015
Open
@dcrankshaw
Copy link
Contributor Author

This is subsumed by #35.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dcrankshaw dcrankshaw

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dcrankshaw