Web Proxy Tools (Fiddler and Charles)

Almost every application these days communicates over HTTP: websites, RESTful services, and even SOAP APIs all make use of Hypertext Transfer Protocol. For the most part, we don’t worry too much about what is happening at the network level when we’re building these applications.

However, from time to time a problem will show up that needs us to drop to the level of the network to figure out what is going on. This is the story of one such situation and how using Fiddler or Charles could make your life much easier.

How to identify HTTP-based issues?

In order to identify HTTP-based issues, the best solution is to capture the data traffic of your app and look at individual requests or even raw packets being sent back and forth. In web-based apps, we would probably look at the data traffic using some kind of inspector, like Chrome DevTools. But what about mobile apps? The best and simplest solution is to use an HTTP debugging proxy server tool.

In general, HTTP debugging proxy server tools can be used for a number of reasons, but they are great for capturing HTTP traffic and logging it for the user to review. Also, most modern tools are capable of decrypting HTTPS (HyperText Transfer Protocol Secure) traffic.

To understand how the decrypting is done, let’s take a closer look at how HTTPS works. In general, HTTPS appears in the URL when a website is secured by an SSL (Secure Sockets Layer) certificate (also known as Transport Layer Security or TLS).

Encounters with SSL usually go like this:

1. The client attempts to connect to a web service secured with SSL.
2. The client requests the web server to identify itself.
3. The server sends the client a copy of its SSL Certificate.
4. The client checks whether it trusts the SSL Certificate. If it does, it sends a message to the server.
5. The server sends back a digitally signed acknowledgment to start an SSL encrypted session.

The way these tools make HTTPS decryption happen is by using self-signed SSL certificates to fake a certificate for the HTTPS endpoint, thus implementing a man-in-the-middle interception. This means that the computer running the proxy tool will be able to see all the data communication that is happening on the mobile device.

Fortunately, there are many great tools available for this purpose, but we will take a look at the two top ones – Fiddler and Charles.

Fiddler

Fiddler is a free web-debugging proxy for any browser, system, or platform that monitors, inspects, edits, and logs all HTTP(S) traffic and issue requests between your computer and the Internet. The tool enables you to inspect incoming and outgoing data, monitor and modify requests and responses before the browser receives them.

Download Fiddler from the Telerik official site.

Putting a debugging proxy between a client and an HTTPS endpoint won’t work because HTTPS cannot be decrypted by the proxy. Fortunately, Fiddler offers a workaround where it will insert a new certificate in the SSL keychain and use that to fake a certificate for the endpoint.

With the SSL certificate in place, network traffic can now be intercepted.

The HTTPS intercept options dialog. Selecting Decrypt HTTPS traffic will insert a new certificate in the certificate chain.

Key Features of FiddlerKey Features of Fiddler

Fiddler Everywhere delivers a range of handy and user-friendly features whose number and further development are not limited.

System Traffic Capturing

Fiddler Everywhere inspects and logs all HTTP(S) and WebSocket traffic between your computer and the Internet and enables to capture traffic from virtually any application that supports a system proxy.

Preconfigured Browser Capturing

Fiddler Everywhere provides pre-configured options for automatic, sandboxed, browser HTTP(S) capturing without the need to modify the operating system proxy.

Traffic Analysis

Fiddler Everywhere provides an HTTP(S) and WebSocket traffic-analyzing functionality, which renders the captured information in a structured way and visualizes various statistical data for one or more captured sessions. It can also decrypt secure traffic and save, receive, and share captured traffic with collaborators.

Modern HTTP version support

Fiddler Everywhere supports the widely used versions of the HTTP protocol, including the widely used HTTP/1.1 and the latest major revision HTTP/2.

Request and Response Mocking

Fiddler supports the mocking and modification of requests and responses on any application. In this way, one can quickly and easily change HTTP(S) requests and responses without the need to update the code.

API Composing

Fiddler Everywhere comes with an API Composer, which allows you to test REST and SOAP APIs by creating and sending requests as well as save and share composed APIs. You can use the API composing feature both when developing new APIs and when testing existing APIs.

Secure Collaboration

Fiddler Everywhere provides a secure method for saving and sharing network debugging logs with your teams for analysis, communication, and resolution.

Cross-Platform Support

Fiddler Everywhere provides platform-agnostic compatibility that enables you to install, configure, and run the web-debugging tool on Windows, macOS, and Linux system-operating machines.

Fiddler Everywhere on Mobile

Fiddler Everywhere enables you to also capture and inspect traffic on devices running on iOS and Android.

Styling Options

Fiddler Everywhere delivers a modern, intuitive UI and a set of popular themes.

Let’s try doing the same analysis using Charles.

Charles

Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables view all of the HTTP and SSL / HTTPS traffic between the machine and the Internet. This includes requests, responses, and the HTTP headers (which contain the cookies and caching information).

Web browser is configured to access the Internet through Charles, and Charles is then able to record and display all of the data that is sent and received.

Since the user is unable to see what is sent and received between browser/client and the server, it is difficult and time-consuming to determine where the fault is without the visibility. Charles makes it easy to determine what is happening, quickly diagnose and fix problems.

Charles makes debugging quick, reliable and advanced, saving you time and frustration!

download the free trial of Charles.

Setting up Charles to handle HTTPS traffic is quite easy. The setting can be found under the Tools menu and is called SSL Proxying Settings.

The options dialog for SSL interception in Charles.

Key Features of Charles

SSL Proxying

Charles can act as a man-in-the-middle for HTTP/SSL communication, enabling you to debug the content of your HTTPS sessions.

Bandwidth Throttling

Charles simulates modem speeds by effectively throttling your bandwidth and introducing latency, so that you can experience an entire website as a modem user might (bandwidth simulator).

AJAX

Charles is useful for XML development in web browsers, such as AJAX (Asynchronous JavaScript and XML) and XMLHTTP, as it enables you to see the actual XML that is flowing between the client and the server. Charles natively supports JSON, JSON-RPC and SOAP, displaying each in a simplified tree format for easy viewing and debugging.

Flash

Charles is great for Adobe Flash developers as you can view the contents of Load Variables, Load Movie and XML loads. Charles also has native support for Flash Remoting (AMF0 and AMF3).

Autoconfigure Browser & System Proxy Settings

Charles will autoconfigure your browser's proxy settings on the following browsers:
Windows system proxy settings (includes Internet Explorer and most other applications).
Mac OS X system proxy settings (includes Safari and most other applications)
Mozilla Firefox (on all platforms)

Other features of Charles.

• Repeat requests to test back-end changes
• Edit requests to test different inputs
• Breakpoints to intercept and edit requests or responses
• Validate recorded HTML, CSS and RSS/atom responses using the W3C validator
• Map Local: Map a Response with a local file or local directory
• Map Remote: Change the request destination to different destinations
• Breakpoint: Set a breakpoint on matched requests or responses and be able to manipulate the content on-the-fl
• Allow / Block List: Allow or block certain requests
• Rewrite / Scripting: Write a script, or define a rule, which is automatically executed at runtime. It's useful for automation work in comparison with the Breakpoint Tool
• External Proxying: Proxy to the external central server (suitable for big companies has a Proxy server)
• Network Throttling: simulate several types of network connections

Comparison of Charles Proxy, Fiddler

In general, those man-in-the-middle apps can be used for:

• Capture and Inspect HTTP/HTTPS traffic in plain text
• Observing or analyzing HTTP(s) Requests and Responses
• Sharing HTTP(s) Requests and Responses
• Capturing traffic from iOS physical devices (iPhone, iPad, Apple Watch, etc.), iOS Simulator, Android physical devices, and Android Emulators.
• Easily manipulating HTTP(s) Requests and Responses data with built-in tools, such as Map Local, Map Remote, Breakpoint, Scripting

There are a number of good options that can fill the role of a debugging proxy. There are many other tools like Wireshark, Debookee, or Proxyman which perform almost the same task.

Each of them has its own advantages/disadvantages which fit the needs according to the task to be performed.

About Encora

Fast-growing tech companies partner with Encora to outsource product development and drive growth. Contact us to learn more about our software engineering capabilities.