Peeking into HTTPS Traffic with a Proxy

[YulianaBoyle]

Peeking into HTTPS Traffic with a Proxy.
In this article, we’ll show you how to configure your web application, Appsmith in this case, to run correctly behind a firewall that does SSL decryption.

This article is about configuring a web application, Appsmith in this case, to run correctly behind a firewall that does SSL decryption, as a Docker container. Instead of a firewall, we’ll use a proxy, which, for the purpose of the problem statement, should be the same.

Table of Contents

Setting up mitmproxy

Setting up

Setting proxy on the whole container

Conclusion

Bonus: Using Charles

Since the proxy needs to support HTTPS decryption, we’ll use mitmproxy, but Charles or any other proxy that supports this would also work just fine.

Setting up mitmproxy

Install with:

brew install mitmproxy

Now launch it using:

mitmweb --listen 9020 --web 9021

Let it run in a separate Terminal window in the background. This will also open the proxy’s web UI at http://localhost:9021. To get a console UI instead, use mitmproxy instead of mitmweb in the above command.

Let’s try running some requests through this proxy to see it’s working well. Start with:

curl http://httpbun.com/get

This should print a valid JSON as the response, with some details about the request itself. Let’s repeat this with the proxy.

curl --proxy localhost:9020 http://httpbun.com/get

You should again see the same response here, but this time, a new entry should appear in the mitmweb UI. Here, you can inspect the request and be able to see the path, headers and response of the request.

So we’ve confirmed that our proxy works. Let’s add HTTPS to the mix.

Again, same thing, but with HTTPS, without a proxy. You should see the same response as before, but without an entry in the proxy. That’s to be expected since we didn’t put a --proxy here. Let’s try that now.

curl --proxy localhost:9020 https://httpbun.com/get

This will fail with a verification error, that the SSL certificate couldn’t be verified.

Let’s see why. The way an SSL proxy works is by establishing two SSL connections, one with the client (a browser, or curl), initiated by the client, and another with the server (the httpbun.com server in this case). Everything sent by the client is encrypted using the certificate of mitmproxy, and everything by and to the server is encrypted with the server’s certificate.

The first time mitmproxy is started, it creates a new root certificate, in the ~/.mitmproxy folder. We can install this root certificate on our system, and then curl, or any other client, will trust it. The mitmproxy docs talk about how to install this cert. Optionally, for curl, instead of installing the cert, we can use the --cacert flag to point to the root certificate.

Another point to note here, is that installing this root certificate on your system, doesn’t mean it’ll be trusted in any Docker containers run on your system. Docker containers are isolated systems in this context, and maintain their own list of trusted root certificates.

To illustrate this, first, let’s run the same request from inside a container, and we should see the error right away:

docker run --rm alpine/curl --proxy host.docker.internal:9020 https://httpbun.com/get

At this, you should see a certificate validation error. This is because the root certificate of mitmproxy isn’t installed inside the container’s environment, and so the curl invocation inside, won’t be able to verify mitmproxy‘s certificate.

To confirm that this is indeed because of mitmproxy, run the same docker run command without the --proxy host.docker.internal and you won’t see this error, despite running with https.

Now we’ve reproduced the situation where a process (a web server in our case), inside a Docker container, is trying to run behind an SSL-decrypting firewall (or, an SSL-decrypting proxy in our case here). Let’s see what we can do to get this to work.

[BrandonHarris]

I found your article quite interesting! It's essential to learn how to configure web applications to work smoothly behind firewalls with SSL decryption, especially when using Docker containers and proxies.

[PitSterw]

Thanks for sharing this informative article about configuring web applications to work behind a firewall that performs SSL decryption. This is a valuable topic, especially in today's security-conscious digital landscape. Proxy solutions can play a significant role in managing and securing traffic, including HTTPS traffic. It's essential for developers and administrators to understand how to properly set up such configurations to ensure the smooth operation of their web applications while maintaining security. I encourage everyone interested in this topic to read the full article on appsmith.com to gain a deeper understanding of the SSL decryption process and how it can be implemented for applications like Appsmith.