Drupal.org Migrates Content and File Delivery to Fastly
16 Nov 2015
We are so stoked to announce our partnership with Fastly. Fastly is now serving up all of our traffic from the *.drupal.org domain Drupal sites and related services.
Drupal.org is big
Drupal.org and its services handle over 1.5 billion requests per month; this is a massive amount of traffic for an open source project.
Every time cron runs the update service on your Drupal site, it talks to updates.drupal.org. Every time you download a copy of Drupal or any projects on Drupal.org, you talk to ftp.drupal.org. (We see over 400,000 downloads of Drupal core in a typical month—way more around DrupalCons and major community sprints.)
Each month we have over 15 million unique pageviews on Drupal.org—by over 2 million unique visitors.
All of those stats are about to rise significantly with the launch of Drupal 8. Drupal 7 caused a 30% bump in traffic when it was released and we expect even more with the launch of Drupal 8.
Fastly is… well… fast
In 2014, we implemented a CDN (content delivery network) for Drupal.org. The impact was immediate. Everything was faster. We met our initial goal of getting Drupal.org sites and services fronted by a CDN.
That CDN solution was a good start, and while it lacked features we didn’t know we needed, it improved our ability to deliver Drupal-generated content as well as packaged projects.
Late in 2014, we were introduced to Fastly. They offered us an opportunity to try out the service for ftp.drupal.org and we haven’t looked back.
Varnish for the win!
You may already use the open source Varnish for your Drupal sites. Varnish is an HTTP accelerator. Drupal.org began using Varnish in 2009 to reduce load on its web servers. It is a powerful cache that likely sits in front of your web origins. Fastly gives us a globally distributed Varnish cache with all of the features we are familiar with.
Fastly hires maintainers of the Varnish project and are helping move it forward. Drupal 8 cache keys are also a straight correllation to Fastly Surrogate Key purging and Fastly’s active involvement is another great example of Fastly working with an open source community (ours!) to build a better experience.
Yay, open source!
Open Source Alliance
Speaking of open source, Fastly has an open source alliance that provides free content delivery to projects like Debian, the MIT Media Lab, the W3C, Memcached, Linux Foundation, and more. They have even open sourced some of their technology stack to make it easier for others to build systems with their tools. We love partnering with these sorts of organizations.
Supporting Technology Partner
Speaking of partnering, Fastly took their partnership a step further and are helping fund a lot of great work on Drupal.org through commitments to sponsor DrupalCons and even the release of Drupal 8. Look for them at upcoming DrupalCons. Thanks Fastly!
How Fastly Works
Fastly’s Global Network consists of Points of Presence (POPs) running Fastly’s custom Varnish software stack. These POPs are spread around the world in strategic locations close to the highest density Internet Exchange Points, ensuring cached content is just a few milliseconds away.
Fastly has a powerful feature that allows us to specify a specific POP as a shield to our origin server. In our case we chose the Seattle POP because of its proximity to our servers at the <a href=”http://osuosl.org”>OSL</a> in Corvallis, Oregon in the United States.
The origin shield configuration means all requests to Drupal.org flow through Fastly’s origin server in Seattle before reaching our Drupal.org origin. If Fastly’s origin server in Seattle has already cached the file, no request to our Drupal.org origin is necessary.
In practice, this allows us to deliver a huge amount of content with an extremely high hit ratio. (See that spike in the GIF below, that’s one of those regular times that cron jobs request a ton of content from our updates server. Fastly is just churning along.)
(Fastly made us confident we could put this enormous GIF in our write up.)
Faster changes, faster response time
Another advantage to Fastly has been the efficiencies it has given us by allowing us to quickly make changes to how they cache our content. With our previous CDN, changes could take up to 4 hours to propagate through the system. We can now change a Varnish config and reflect those changes in under 5 seconds. That has been a huge help when we need to make a hot fix to production and keep Drupal.org delivering Drupal to the world.
Logging and improving our usage stats
Fastly also allows us to stream our logs. This means we can parse those logs and turn them into data. We are already seeing much more reliable project usage statistics with from our new log processing for updates.drupal.org thanks to Fastly’s log streaming.
More accurate logs mean we can provide users with better data to help them to make better decisions about modules, themes and distributions that have the most installations.
Next Steps with Fastly
We have been impressed with Fastly, both for their responsiveness and the performance improvements we have seen while using their services. They have given a ton to our community and we can’t wait to see where they take Varnish and their open-source-powered service. If you are interested in using Fastly with your own Drupal sites, you can learn more about Fastly on their Drupal.org organization page or sign up for a developer account at Fastly.com—there is even a Fastly module for Drupal7, and one for Drupal8 is on the way!