A few days into the attack, our “Network Fortress” plan had already changed.
We had started with the idea of using GRE tunnels for the production data center. Then we studied what had happened to other large internet services and saw the weakness: a committed attacker could find the tunnel endpoint and attack the origin directly.
The design needed to evolve with the threat.
For the production site, we decided to keep traffic behind Prolexic’s proxy and allow only tightly controlled traffic from the mitigation network through our firewalls. A GRE tunnel could still provide additional aggregate bandwidth if necessary, but it would not be the long-term foundation.
The larger move was private connectivity. We wanted highly available fiber directly into Prolexic facilities or a connection through the private MPLS ring we were already building between data centers. If we removed the public origin from the path, an attacker had less to discover and target.
We were planning to defend two to three gigabits of attack traffic in aggregate. The full integration would take months because fiber had to be built and the data-center ring was still in progress. After that, the office could connect over dark fiber and stop depending on its temporary tunnel.
This was not a security appliance purchase. It was a change to the shape of the network.
The attack also changed the operating plan. We would need capital sooner, a continuing service commitment to the mitigation provider, and another network hire in 2014. The resulting private mesh would be more capable and more complicated. Somebody had to own that complexity.
I was still away from New York while working the deals and implementation. I wrote that I would return once I had confidence in the initial security. That line captures how consuming the response had become.
The useful lesson was not that our first design was wrong. It was that we changed it when new evidence arrived. We compared notes from other attacks, looked for the point where their defenses failed, and rebuilt our approach around a harder origin to reach.
The attacker was adapting. We had to adapt faster.