{"id":606,"date":"2016-05-19T07:15:14","date_gmt":"2016-05-19T12:15:14","guid":{"rendered":"http:\/\/blog.tiingo.com\/?p=273"},"modified":"2026-07-22T14:18:31","modified_gmt":"2026-07-22T18:18:31","slug":"switched-away-aws-packet-net-benchmarking-networking-disk-processing-speeds","status":"publish","type":"post","link":"https:\/\/www.tiingo.com\/blog\/switched-away-aws-packet-net-benchmarking-networking-disk-processing-speeds\/","title":{"rendered":"AWS vs Packet.net: Why We Left AWS & Benchmarking AWS’s Network, Disk, and CPU Performance"},"content":{"rendered":"

If this sounds like a glowing review of Packet.net<\/a> – it is. I found myself re-reading this post over and over, trying to make it sound less shrilly – but I can’t. It’s just a ridiculously good product and value – EC2 containers just don’t make sense anymore.<\/p>\n

A friend once told me, “Rishi – sometimes if you don’t advocate\u00a0a product aggressively – you can be doing society a disservice in your attempt to be neutral. If the value is so good, you\u00a0must<\/em> tell everybody about it.”<\/p>\n

This is one of those times.<\/p>\n

EDIT:<\/b>\u00a0Feeling really grateful the HackerNews community decided to link to Tiingo a second time. In the first HackerNews posting many of you asked for an API, which is what led to me finding the AWS bottleneck. The API launched [quietly] this week at:\u00a0https:\/\/api.tiingo.com<\/a> where Tiingo is now the first company to bring IEX (anti-HFT exchange\/darkpool) data to mainstream Financial Tech firms. Kind of went full-circle as this post wouldn’t have existed without the original HN coverage<\/a>.<\/p>\n

\n

TL;DR:<\/strong><\/h2>\n

The performance of AWS on network speed, disk speed, and CPU performance are quantitatively just “not good,” for what we needed. When we introduced real-time market data, we were in search of our bottleneck and realized it was AWS. We made the decision to switch to Packet.net<\/a>\u00a0and the below reflects on our decision and explains why.\u00a0The benchmarks continue to reaffirm our decision.\u00a0Having said all of this, certain features of AWS\u00a0remain incredibly convenient like S3, Cloudfront, and Route53 – but we can’t justify\u00a0using EC2.<\/p>\n

In Networking<\/strong>: Packet is\u00a0significantly faster,<\/em> more stable,\u00a0and 15%-44% cheaper<\/p>\n

In Disk Usage<\/strong>: Packet is more performant and 92% cheaper<\/p>\n

In CPU:<\/strong> Packet is 30-40% more performant and 15% cheaper<\/p>\n

In machines:<\/strong> Packet’s systems are all bare-metal\/dedicated, whereas AWS charges extra\u00a0for dedicated machines<\/p>\n<\/div>\n

If you’ve noticed Tiingo being particularly snappy these days, it’s because I couldn’t stand it anymore. I had tried everything – buying more expensive instances on AWS, allocating more space, scaling horizontally, but it wasn’t matching up to my local dev machines. And so I started searching for the bottleneck – only to realize it was AWS.<\/p>\n

I started researching AWS, I found I wasn’t alone. Many people experienced what I had but I tried prolonging the switch. Trying to change cloud service providers is \u00a0frustrating: scripts break, performance temporarily suffers, you experience downtime, and you know there will be unknown-unknowns.<\/p>\n

Recently we just got real-time market data and this exacerbated the issues. Our websockets were being overwhelmed in queues and throwing more nodes at the problem was becoming expensive. We were\u00a0trying to put a bandaid over a burst pipe. I\u00a0finally decided on\u00a0Packet.net<\/a>\u00a0and I want to share the reasons why. I’ve included\u00a0benchmarking results to help emphasize the point.<\/p>\n

Our search was motivated by\u00a0two<\/strong> major reasons:<\/p>\n

    \n
  1. The costs were getting out-of-hand<\/li>\n
  2. After reading the below Reddit post on AWS’s [lack of] network stability, we started asking around and realized the experts were right… AWS’s network is slow. If we are\u00a0going to give our users real-time data directly from the exchanges, that’s a heck-of-a-lot of data and we need it to be as fast as possible.<\/li>\n<\/ol>\n

    The\u00a0Reddit\/Blog Post was from an engineer at Stack Overflow.<\/p>\n

    http:\/\/nickcraver.com\/blog\/2016\/02\/17\/stack-overflow-the-architecture-2016-edition\/<\/a>.<\/p>\n

    More specifically, this Reddit comment on AWS’s network stability that seemed echo’d by many:<\/p>\n

    \"Reddit<\/a>
    https:\/\/www.reddit.com\/comments\/468p2m<\/a><\/figcaption><\/figure>\n

    We explored options like DigitalOcean<\/a>, but Tiingo, like all financial data\/analytics companies, is very data heavy and their plans didn’t allow for flexible data storage (EBS on Amazon for example). We looked into Rackspace and Azure, but the cost differentials didn’t make it seem worth the transition. Admittedly, having used Rackspace in the past – I’ve always loved their customer support and was personally disappointed I couldn’t justify the cost.<\/p>\n

    Eventually I came across Packet and spoke to their engineers since I hadn’t heard of them before.<\/p>\n

    I took a chance. It paid off.<\/p>\n

    I told them my concerns and what I was trying to solve (market data connectivity and high data transfer rates). One of the co-founders, who was a networking engineer,\u00a0personally<\/em> oversaw my connectivity project to the data exchanges. I’m pretty sure this was Paul Graham 101 on start-ups and customer service.<\/p>\n

    Ultimately though – I’m a data nut and so I decided to benchmark AWS vs Packet and was really curious about the Reddit comments on AWS’s network stability. The benchmarks closed the deal for us. It was a no-brainer. Part of the major reason being that Packet.net is bare metal (dedicated physical machines) whereas AWS tends to be focused on virtual machines. The hardware\/pricepoint is actually even cheaper on Packet. We are paying 1\/3rd of what it would cost to get a similar, less performant, system on AWS.<\/p>\n

    SO here you have it!<\/p>\n

    The tests below compare AWS vs Packet for disk, network, and CPU benchmarking – and also cost.<\/p>\n

    I’ve outlined and commented the results below so you can reproduce the tests.<\/p>\n

    Hardware<\/h2>\n

    Since we are testing Packet vs AWS, we started off with the Packet hardware and found the AWS price equivalent. We started with the Type 1 and worked backwards to find the equivalent in performance\/price on AWS.<\/p>\n

    Note: For the network test, we also test a smaller machine. The reason for the lighter hardware is for load balancing (HAProxy in this sense). If all of the back-end servers can have high network throughput, but we need to send it to the end-user, the load-balancer’s networking performance will be the determining factor. This is especially important in cases like real-time data.<\/p>\n

    Packet:<\/strong><\/p>\n\n\n\n\n\n\n
    Instance Name<\/th>\nCPU<\/th>\nMemory\/RAM<\/th>\nPrice<\/th>\nPrice\/Month<\/th>\nNotes<\/th>\n<\/tr>\n<\/thead>\n
    Type 1 (Server)<\/td>\n(4 core, 8-threaded) \u00a03.4ghz Intel Xeon E3-1240 v3<\/td>\n32gb<\/td>\n$0.40\/hr<\/p>\n

    $0.37\/hr\u00a0if reserved for 1 month<\/td>\n

    		$292.80\/month<\/td>\n<\/td>\n<\/tr>\n
    Type 0 (Load Balancer)<\/td>\n(4 core) 2.4ghz Intel Atom C2550<\/td>\n8gb<\/td>\n$0.05\/hr<\/p>\n

    $0.0459\/hr\u00a0if reserved for 1 month<\/td>\n

    		$36.60\/month<\/td>\nWhat somebody may choose as their load balancer<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    *Note:We assume 732 hours in a month; but if you reserve a Packet instance for a month, they will only charge you 672 hours per month. However, to make apples-to-apples comparisons, all calcs in Price\/Month assume you choose hourly pricing (732 hours for 1 month) to keep things normalized.<\/p>\n

    AWS:<\/strong><\/p>\n\n\n\n\n\n\n
    Instance Name<\/th>\nCPU<\/th>\nMemory\/RAM<\/th>\nPrice<\/th>\nNotes<\/th>\n<\/tr>\n<\/thead>\n
    m4.2xlarge (Server)<\/td>\n8 VCPU\u00a0\u00a0(2.4ghz Intel Xeon E5-2676)<\/td>\n32gb<\/td>\n$0.479\/hr<\/td>\n$350.63\/month<\/td>\nxlarge was chosen for it’s optimized network performance<\/td>\n<\/tr>\n
    t2.medium (Load Balancer)<\/td>\n2 VCPU \u00a0(Xeon processors burstable to 3.3ghz)<\/td>\n4gb<\/td>\n$0.052\/hr<\/td>\n$38.07\/month<\/td>\nWhat somebody may choose as their load balancer<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    OS<\/strong>:<\/p>\n

    Ubuntu 14.04 server<\/p>\n

    The Benchmarks<\/h1>\n

    Network:<\/h2>\n

    For this test, we used iperf3 as per the AWS documentation<\/p>\n

    (https:\/\/aws.amazon.com\/premiumsupport\/knowledge-center\/network-throughput-benchmark-linux-ec2\/<\/a>)<\/p>\n

    We wanted to simulate a very real-world network configuration for ourselves – basically what our site looks behind a load balancer. Load balancers tend to require very low processing power, and serve as a network bottleneck to the user.<\/p>\n

    We are testing:<\/p>\n