CVE-2017-7781/CVE-2017-10176: Issue with elliptic curve addition in mixed Jacobian-affine coordinates in Firefox/Java

tl;dr Firefox and Java suffered from a moderate vulnerability affecting the elliptic curve point addition algorithm that uses mixed Jacobian-affine coordinates where it can yield a result POINT_AT_INFINITY when it should not.

Introduction

Few months ago I was working on a vulnerability affecting the internet standard JWE (slides here) and I got a stroke of luck. Yuppieeee  Basically I was constructing the malicious JWEs needed for the Demo Attack. When something weird happened :S
You can try and share with me the surprise I had, the gist is here

If you try to execute this class with Java 1.7 you basically have

Exception in thread "main" java.lang.IllegalStateException
    at sun.security.ec.ECDHKeyAgreement.deriveKey(Native Method)
    at sun.security.ec.ECDHKeyAgreement.engineGenerateSecret(ECDHKeyAgreement.java:130)
    at javax.crypto.KeyAgreement.generateSecret(KeyAgreement.java:586)
    at orig.EccJava.getAgreedKey(EccJava.java:53)
    at orig.EccJava.main(EccJava.java:44)

😲 Wait, what? Ok I know, obviously not clear. Let's step back and slowly move forward.

Invalid curve attack on elliptic curve

In order to understand what is going on here you need to be knowledgeable about elliptic curves and invalid curve attack on them. I tried to give some explanation in the already mentioned post.
Said that let come back to the gist above. Why so much surprise about this java.lang.IllegalStateException ?
As mentioned, in order to exploit the JWE vulnerability present in many libraries, I was crafting malicious JWEs. One of the steps involved to construct an invalid curve somehow related to the famous P-256 curve. One of the malicious curve I came out with for the demo attack had the really low order of 2447.  Hence the attack required me to build 2447 malicious JWEs something like:

G = base point of the invalid curve;
for (i = 1; i<2447; i ++) {
    P = i * G; 
}

All was going pretty well until arrived to the point 2417 (this is basically the gist above) in the loop  BOOM:

Exception in thread "main" java.lang.IllegalStateException

This happened back in March while I was working on the JWE's disclosure. I was extremely surprised to see this Exception!! Why an apparently innocuous normal scalar multiplication was throwing this Exception??? This made me really really curious but unfortunately I did not have time to explore this more deeply. So I simply decided to temporary park it and come back to it having more time.

Elliptic curve point addition in mixed Jacobian-affine coordinates

So once the disclosure was out and after taking few week of rest I was ready to dig deeper this issue. First thing I did was to download the OpenJDK source code and started inspecting. 
After quite a bit of investigation I ended up here:

At the same time I found out that for some reason NSS (hence Firefox) shares the same code base with OpenJDK (for elliptic curve cryptography). Then I found this:

Continue surfing through the code looking for the usage I found the algorithms used were taken from:

/* Computes R = nP where R is (rx, ry) and P is the base point. Elliptic
 * curve points P and R can be identical. Uses mixed Modified-Jacobian
 * co-ordinates for doubling and Chudnovsky Jacobian coordinates for
 * additions. Assumes input is already field-encoded using field_enc, and
 * returns output that is still field-encoded. Uses 5-bit window NAF
 * method (algorithm 11) for scalar-point multiplication from Brown,
 * Hankerson, Lopez, Menezes. Software Implementation of the NIST Elliptic
 * Curves Over Prime Fields. */

Cool. Let's look at this Brown et al paper.  Here it is:

Oh boy but this is exactly what is implemented in the code so what is wrong? All seems legit... :S Hold on a sec, what is happening if C = X2 Z1^2  -  A is equal to zero ? It must be something wrong here... Of course it is. Got it, the if/else block is missing as per here

Holy Elliptic Curve Batman, so instead of doubling a point the algorithm returns POINT_AT_INFINITY!!

Exploitability

OK, now we know what is wrong, but what is special about 2417? Why this is not happening with 2416 or 1329 instead? The reason why the issue is triggered goes along this lines:

Basically at a certain point of the algorithm (toward the end, remember it uses 5-bit window NAF) needs to do 2432 -15 = 2417 . Now 2432 = -15 mod (2447) so ==> -15 -15 and BOOOM rather than double the point the algorithm returns point at infinity....!!

Right. Next natural question: is this something special about this invalid curve* or this can be reproduced with any curve (e.g. a standard curve)?

With another bit of tweaking I came out with a P-521 example (hence even the last patched version of Java was affected):

Cool, and now? Can we exploit this in some way and recover any private key? The reality is PROBABLY NOT :( Believe me, I have tried hard to exploit this stuff but nothing, niente, nada, niet, nisba! At least for the classic ECDH where the private key is not under attacker's control. Maybe this could be exploited if this very same code is employed to implement some other more "exotic" protocol (e.g. PAKE) ? In any case, the comment section of the blog is open and my Twitter DM is open, so hit me up if you have any idea.....

Disclosure timeline

Apr-2017 - Reported to Mozilla security team.

Apr-2017 - Reported to Oracle security team.

Jul-2017 - Oracle Critical Patch Update Advisory - July 2017 (CVE-2017-10176), fix here

Aug-2017 - Mozilla Foundation Security Advisory 2017-18 (CVE-2017-7781), fix here

Acknowledgement

I would like to thank the Oracle and Mozilla team for the constant and quick support specially to Franziskus Kiefer.

That's all folks. For more crypto goodies, follow me on Twitter.

 *  Java SUN JCA provider that comes with Java later than version 1.8.0_51 are not affected by invalid curve attacks since they check for point on the curve.