#2086 new
Vadim Manikhin

ConcurrentModificationException while storing data before Continuation suspend

Reported by Vadim Manikhin | March 15th, 2017 @ 12:54 PM

Framework version: 1.4.4
Platform you're using: Linux Debian 8

Reproduction steps: happens very occasionally (2 times in last 7 days)

Details:

Exception: play.exceptions.JavaExecutionException Caused by: java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) at java.util.HashMap.putAllForCreate(HashMap.java:554) at java.util.HashMap.(HashMap.java:298) at play.mvc.Controller.storeOrRestoreDataStateForContinuations(Controller.java:1172) at play.mvc.Controller.await(Controller.java:1218) at controllers.api.ItemAPI.update(ItemAPI.java:370) at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:536) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:473) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:467) at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:436) at play.mvc.ActionInvoker.invoke(ActionInvoker.java:159) at Invocation.HTTP Request(Play!)

ItemAPI.update:

Object result = await(new Job() {
@Override public Object doJobWithResult() { ... } }.now());

Comments and changes to this ticket

  • David Costanzo

    David Costanzo May 2nd, 2018 @ 08:42 PM

    This is due to a race condition in F$Promise.onRedeem() that can cause the callback that is being registered to be called twice. This can have a variety of pathological behavior on a Play applications, particularly when a controller action invokes a job that finishes quickly.

    1) await(Future) "returns" twice (in two different threads)
    2) After await(Future), some controller action parameters are null.
    3) In await(Future), there's a ConcurrentModificationException whose stack resembles

    java.util.ConcurrentModificationException
        at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
        at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752)
        at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750)
        at java.util.HashMap.putMapEntries(HashMap.java:511)
        at java.util.HashMap.<init>(HashMap.java:489)
        at play.mvc.Controller.storeOrRestoreDataStateForContinuations(Controller.java:1172)
        at play.mvc.Controller.await(Controller.java:1218)
    

    The problem is an incorrect usage of synchronization in F$Promise.onRedeem() and F$Promise.invokeWithResultOrException().

    public void onRedeem(F.Action<Promise<V>> callback) {
        synchronized (this) {
            if (!invoked) { <-- assume invoked = false, so we add callback to the callbacks list
                callbacks.add(callback);
            }
        }
        // <-- context switch here to invokeWithResultOrException
        //     When control returns here, it's possible that invoked==true and the
        //     callback has already been called.
        //     In this case, callback should NOT be called again.
        if (invoked) {
            callback.invoke(this);
        }
    }
    
    protected void invokeWithResultOrException(V result, Throwable t) {
        synchronized (this) {
            if (!invoked) {
                invoked = true;
                this.result = result;
                this.exception = t;
                taskLock.countDown();
            } else {
                return;
            }
        }
        for (F.Action<Promise<V>> callback : callbacks) { // <-- callbacks is not protected by synchronized, so it could be modified during iteration
            callback.invoke(this);
        }
    }
    

    This causes rare failures in recommended use-await-to-avoid-blocking-controller-action pattern, but the failures are more frequent for controller actions that use chunking and get the contents of each chunk in a separate Job invocation.

    I intend to submit a patch for this on GitHub, but even before this is fixed in Play!, the effects from await(Future) can be fixed in application code by adding a patched Job subclass whose .now() returns a patched Promise and then deriving all of your Job classes from that class. Here is the class which I am using.

    public abstract class FixedJob<V> extends Job<V> {
        /**
         * A fixed form of Promise. This is the class which has the bug.
         */
        private static class FixedPromise<V> extends Promise<V> {
            @Override
            protected void invokeWithResultOrException(V result, Throwable t) {
                List<F.Action<Promise<V>>> callbacksWhenLocked = null;
    
                synchronized (this) {
                    if (!invoked) {
                        invoked = true;
                        this.result = result;
                        this.exception = t;
                        callbacksWhenLocked = new ArrayList(callbacks);
                        taskLock.countDown();
                    } else {
                        return;
                    }
                }
                for (F.Action<Promise<V>> callback : callbacksWhenLocked) {
                    callback.invoke(this);
                }
            }
    
            @Override
            public void onRedeem(Action<Promise<V>> callback) {
                boolean mustCallCallback;
                synchronized (this) {
                    if (!invoked) {
                        callbacks.add(callback);
                    }
                    mustCallCallback = invoked;
                }
                if (mustCallCallback) {
                    callback.invoke(this);
                }
            }
        }
    
        /**
         * Start this job now (well ASAP)
         *
         * @return the job completion
         */
        @Override
        public Promise<V> now() {
            Promise<V> smartFuture = new FixedPromise<>();
            JobsPlugin.executor.submit(getJobCallingCallable(smartFuture));
            return smartFuture;
        }
    
        private Callable<V> getJobCallingCallable(final Promise<V> smartFuture) {
            return new Callable<V>() {
                @Override
                public V call() throws Exception {
                    try {
                        V result = FixedJob.this.call();
                        if (smartFuture != null) {
                            smartFuture.invoke(result);
                        }
                        return result;
                    } catch (Exception e) {
                        if (smartFuture != null) {
                            smartFuture.invokeWithException(e);
                        }
                        return null;
                    }
                }
            };
        }
    }
    
  • Play Duck
  • Play Duck

    Play Duck October 12th, 2018 @ 07:08 AM

    (from [9f3250a6b8b9afb7f64210040d8ea0753ae14595]) Merge pull request #1241 from davidcostanzo/lighthouse-2086-patch

    [#2086] Fix race in Promise.onRedeem() that called "callback" twice https://github.com/playframework/play1/commit/9f3250a6b8b9afb7f6421...

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2>Play framework</h2>

Play makes it easier to build Web applications with Java. It is a clean alternative to bloated Enterprise Java stacks. It focuses on developer productivity and targets RESTful architectures. Learn more on the <a href="http://www.playframework.org">http://www.playframework.org</a> website.<br><br>

<h2>Source code is hosted on github</h2>Check out our repository at <a href="http://github.com/playframework/play">http://github.com/playframework/play</a><br><br>

<h2>Contributing, creating a patch</h2> Please read the <a href="http://play.lighthouseapp.com/projects/57987/contributor-guide">contributor guide</a><br><br>

<h2>Reporting Security Vulnerabilities</h2> Since all bug reports are public, please report any security vulnerability directly to <em>guillaume dot bort at gmail dot com</em>.<br><br>

<h2>Creating a bug report</h2> Bug reports are incredibly helpful, so take time to report bugs and request features in our ticket tracker. We’re always grateful for patches to Play’s code. Indeed, bug reports with attached patches will get fixed far quickly than those without any.<br><br>

Please include as much relevant information as possible including the exact framework version you're using and a code snippet that reproduces the problem.<br><br>

Don't have too much expectations. Unless the bug is really a serious "everything is broken" thing, you're creating a ticket to start a discussion. Having a patch (or a branch on Github we can pull from) is better, but then again we'll only pull high quality branches that make sense to be in the core of Play.

Pages