Security experts are probably familiar with the “defense in depth” concept in which multiple layers of security controls (defense) are placed throughout an IT system. The “defense in depth” approach generally aims to provide redundancy in the event that a security control fails or a vulnerability is exploited.
Network measurement testing conducted via Raspberry Pis deployed in Egypt suggests that ISPs appear to be applying a “defense in depth” strategy for networking filtering, particularly in relation to the blocking of fj-p.com: the site of Egypt’s Freedom and Justice Party (FJP).
When accessing http://www.fj-p.com, a user is redirected to http://fj-p.com which is served from an unknown location via a CloudFlare CDN. Both http://www.fj-p.com and http://fj-p.com are blocked, but they appear to be blocked by different firewall implementations. We have previously identified a middlebox in Egypt, fingerprinted with IPID 0x1234. Our latest testing, though, shows that there is also another middlebox, fingerprinted with IPID 0x0000. Both middleboxes are located within the same Egyptian network, but the latency to those middleboxes is slightly different: 0x1234 is ~33ms away, while 0x0000 is closer at ~30ms away.
The following traceroute does not explicitly highlight the exact IP addresses of the middleboxes, but helps to understand the possible route of the “censored” HTTP request through the network.
HOST: lepidopter Loss% Snt Last Avg Best Wrst StDev
1. AS??? 10.x.y.z 0.0% 10 0.6 0.6 0.6 0.7 0.0
2. AS??? 10.x.a.b 0.0% 10 1.0 0.9 0.9 1.0 0.0
3. AS??? ??? 100.0% 10 0.0 0.0 0.0 0.0 0.0
4. AS20928 bng.rams.ca (22.214.171.124)
0.0% 10 38.8 33.9 29.7 38.8 3.1
5. AS??? 172.17.51.73 0.0% 10 116.5 44.1 32.4 116.5 25.6
6. AS2914 126.96.36.199 0.0% 10 65.8 66.5 63.6 70.1 1.7
When comparing these two middleboxes, 0x1234 does not set DF (Don’t fragment) field of the IP header of TCP RST packet, but 0x0000 does. The 0x1234 middlebox seems to have an initial IP TTL of 64 (the client sees 59), while 0x0000 has an IP TTL of 32 (the client sees 28). 64 and 32 are assumptions based on the fact that TTL values are usually aligned with the power of 2. Hop distance is well-aligned with extra latency (0x1234 ~ 5 hops ~ 33ms, 0x0000 ~ 4 hops ~ 30ms).
The raw TCP window size value is also static and different between the two middleboxes. 0x1234 has a raw window size value of 32120, while 0x0000 is 229. The window size value does not matter for RST packet, but it’s a mandatory field of TCP header.
The 0x1234 RST packet has no payload, while the 0x0000 RST packet has 22 zero-bytes (that’s not “Ethernet padding”, but real zero bytes according to the length field of IP packets). Moreover, sending those two HTTP requests to a random IP address (e.g. one of Google’s IPs: 188.8.131.52) also triggers two different firewalls/middleboxes having a slightly different configuration. The experiment of sending two unrelated HTTP requests to a single Google IP address is done to ensure that these two middleboxes interpreting the HTTP protocol are on the same network path from the client’s IP to the server’s IP, and that that’s not just two distinct middleboxes on two distinct paths.
All of the above suggests that there are two different middleboxes doing the filtering on that specific path, triggered by slightly different rules: a request to http://www.fj-p.com triggers the 0x0000 middlebox, while a request to http://fj-p.com triggers the 0x1234 middlebox.
This appears to be a “defense in depth” strategy applied to censorship, confirmed by doing a TCP traceroute with HTTP payload. A request to http://www.fj-p.com (that triggers the 0x0000 middlebox that is closer) receives RST since the client’s TTL=5, while http://fj-p.com (that triggers the 0x1234 middlebox, is ~3..4ms further away, and that has a hypothetical reverse-path of one hop longer) receives RST since the TTL=6 (as confirmed by a small series of experiments).
Moreover, the 0x0000 middlebox does not appear to be able to reassemble HTTP requests in some cases. Some extra testing with http://www.fj-p.com shows that when the HTTP verb (e.g. `GET`) is split in the middle (like `GE || T …`), there is no usual RST at TTL=5 from 0x0000, but there is RST with TTL=6 from 0x1234. This strongly suggests that these two middleboxes behave differently when the HTTP verb is split.
Both middleboxes, however, appear to reassemble split `Host` headers. When splitting the domain in the middle, `Host: www.f || j-p.com` triggered the 0x0000 middlebox, while `Host: fj-p. || com` triggered the 0x1234 middlebox (as expected).
In short, both middleboxes replaced the original requests with RST packets when forwarding them to servers (since the TTL field set by the client was preserved while forwarding the packet). This suggests that both middleboxes are “in-path” (man-in-the-middle), not “on-path” (man-on-the-side), leading us to think that Egyptian ISPs are applying “defense in depth” tactics for network filtering.
The traffic between Cloudflare’s Point-of-Presence in Cairo and the backend servers of sites using Cloudflare (which are located outside of Egypt) appears to be filtered.
Hundreds of OONI Probe network measurements presented Cloudflare-specific errors (such as 525, which occurs when the SSL handshakes to Cloudflare fails), suggesting that Egyptian ISPs are blocking sites that use Cloudflare by interfering with SSL encrypted traffic between the Cloudflare CDN and the website backend.
We excluded unencrypted HTTP sites that presented such anomalies (such as zenvpn.net and tunnelbear.com) in case the channel between the client and Cloudflare was tampered with, and limited our findings to encrypted HTTPS sites (enabling us to confirm their blocking with more confidence). This left us with circumvention tool sites psiphon.ca and purevpn.com, and news website ultrasawt.com, which appear to be blocked by some form of network interference on the SSL connection between the websites’ backend servers and a Cloudflare CDN in Egypt.
This particular type of network interference could either be attributed to a man-in-the-middle attack on the SSL encrypted connection between Cloudflare and the website backend (as suggested by data from other tests), or to a man-on-the-side attack to terminate or interfere with the SSL handshake.
All of the measurements pertaining to these cases are available here.