Mapping AWS data centers for fastest connection | TurnKey GNU/Linux
You are here
Category: All
Mapping AWS data centers for fastest connection
Blog Tags:
s3
aws
ec2
amazon
Mapping AWS data centers for fastest connection
Alon Swartz
- Thu, 2011/12/29 - 14:25 -
19 comments
Yes, that's 'fastest', not closest.
Background
A while back I published a blog post entitled
Finding the closest data center using GeoIP and indexing
, which described how we automatically determine the AWS regional data center to be used for storing encrypted
server backups
We used the same solution to determine the preferred region when launching
cloud servers
in the Hub, as well as selecting the
closest APT package archive
for all TurnKey deployments.
New and improved
Since the original publication, Amazon built new regional data centers in Oregon, Sao Paulo and Tokyo, so the indexes needed to be updated.
While adding support for the new regions I decided to take it a step further and add some improvements.
Improvement #1: Automatic association (distance)
The method originally used to perform automatic association of countries/states to data centers was lacking some what and needed to be improved.
We are now using the
Haversine formula
, which is used to determine great-circle distances between two points on a sphere from their longitudes and latitudes.
Improvement #2: Incorporated world wide underwater cables (latency)
Originally we relied on user feedback of connection latency to tweak the indexes. This didn't scale very well, so we needed a way to make it easier.
Based on
Gregs Cable map
, we could mashup the automatic associations and tweak the index overrides based on expected latency.
It turns out that this was a crucial part of the equasion, as a user might be physically closer to data center X, but in reality the connection to data center Y is faster. For example, previously Australia was allocated to Singapore but has been moved to California as the pipe is much fatter (see the visual map below).
Improvement #3: Open source
We originally published the indexes, but have now open sourced the whole project on
github
in hope that others might find it useful, and make collaboration easier.
Putting it all together
The below screenshot plots countries/states to their associated AWS regional data centers, and overlays the world wide underwater cables for reference:
Want to zoom in? Toggle active and future cables? Check out the
live mashup
Comments
How do you measure latency to different AWS datacenters?
Liraz Siri
- Sun, 2012/01/01 - 06:15
I agree that it would be better to just go ahead and measure latency and bandwidth directly.
The trick is how do you do that from all over the world. Hmmm... maybe if the Hub doesn't already have your network's routing information it asks the appliance to run a few tests and measure latency directly, then caches that information for future reference.
reply
Global routing expert needed
Liraz Siri
- Sun, 2012/01/01 - 06:17
Good thing Alon open sourced the code. Now let's hope a global routing expert runs into it and decides it would be a fun way to spend some time off on the weekend.
reply
Interesting
Chris Musty
- Fri, 2011/12/30 - 05:05
Above comments aside, I actually had no idea Virginia would be faster than Singapore (I am in Sydney Australia). I have never had need to test for speed issues because Singapore is fast enough for the db frontend apps I am developing. If I can get it faster in Virginia - SWEET! The only issue with that is when the client asks "where is the cloud", sometimes they get turned off when you say "another country".
So as far as practical use goes it may help with data heavy apps in the future, at least for my company.
I am just about to start testing cloud servers and backups with my apps after spending months testing on site installations with TKL (which all came up trumps incidentally) so this information comes with good timing.
Thanks guys!
Chris Musty
Director
Specialised Technologies
reply
Gotta love it!
Chris Musty
- Wed, 2012/01/11 - 13:10
Just how I like my networking cheap and fast!
Chris Musty
Director
Specialised Technologies
reply
Good catch
Alon Swartz
- Fri, 2012/01/13 - 08:43
Good catch Jack - thanks, it was a typo - should have been California. I've updated the blog post.
reply
Sure Eric, go ahead.
Alon Swartz
- Wed, 2012/07/11 - 09:36
Sure Eric, go ahead. All I ask is that you link back to the original article.
reply
It is "current" for TurnKey purposes
Jeremy Davis
- Sat, 2015/11/07 - 08:28
But seeing as AWS have added Frankfurt (which we are yet to support) it could not be considered up to date in that sense...
reply
Pages
Add new comment
Apps
Specials
Web development
Framework
Stack
IT Infrastructure
Content management
Blogging
Ecommerce
Education
Wiki
Media
Business management
CRM
ERP
Invoicing
Messaging
Email
Forum
Chat
Issue tracking
Project management
Database
NoSQL
Developer tools
Help
Forums
Support
General
Development
Documentation
Security and News Announcements
Low-traffic newsletter: up to one email a month.
Previous issues
Categories
development
news
appliances
community
debian
release
hub
stable
iso
security
cloud
ec2
aws
proxmox
lxc
ubuntu
tkldev
tips
v16.x
drupal
More tags
Recent posts
Free up disk space
4th Aug, 2024
Python PEP 668 - working with "externally managed environment"
29th Jul, 2024
v18.0 Stable Release #6 - 10 Newly Updated ISOs, Hub Builds & Proxmox/LXC builds
17th Apr, 2024
v18.0 Stable Release #5 - 20 Updated ISOs & Hub Builds - Proxmox/LXC builds
12th Feb, 2024
v18.0 Stable Release #4 - 10 Updated ISOs & Hub Builds - Proxmox/LXC builds
5th Feb, 2024
1 of 63
next ›
Archive
August 2024
(1)
July 2024
(1)
April 2024
(1)
February 2024
(2)
November 2023
(1)
October 2023
(1)
September 2023
(1)
July 2023
(1)
April 2023
(1)
March 2023
(1)
December 2022
(1)
November 2022
(2)
Pages
Recent comments
V19?
TurnKey LXC templates are available via the Proxmox UI
Non-Aws platforms
Thanks for that!
the command for adding the
mtoolshub
Great post