The Observer methodology
Why the Observer ?
Since more than 10 years, the Cloud Mercato's platform runs network performance evaluation on the cloud infrastructures of the all the competitors in our radar. Out tests involves intra-datacenter and inter-datacenter links performance, therefore mainly from server to server.
While these values are also very interesting, we lack from the measurement of "real" client connections. Cloud providers are generaly located in the same building or at least location linked by low-latency connection. We needed a tool capturing the performance from users from classic ISPs and avoid to navigate only on the Internet backbone.
The Observer is born from this requirement and fill the following purposes:
- Test the connection from anywhere
- Allow any kind of equipement
- Allow anyone to capture his/her performance
Our test network
The Observer's performance value is mainly based on HTTP, so we decided to rely on the different endpoints given by the vendors. Below, you'll find a live map of our probes' locations:
{"scope": "world", "projection": "equirectangular", "height": 500, "width": null, "responsive": true, "fills": {"defaultFill": "#4baf50", "azure": "#55b3ff", "oracle": "#FF0000", "scaleway": "#510099", "linode": "#00b64c", "bso": "#a84091", "oci": "#2E54A4", "aws": "#f7981d", "ovh": "#484848"}, "dataType": "json", "data_type": null, "center": null, "scale": null, "data": {}, "archs": [], "bubbles": [{"name": ["West Europe"], "latitude": 52.3667, "longitude": 4.9, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Netherlands", "flag": "\ud83c\uddf3\ud83c\uddf1"}, {"name": ["North Europe"], "latitude": 53.3478, "longitude": -6.2597, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Ireland", "flag": "\ud83c\uddee\ud83c\uddea"}, {"name": ["UK South"], "latitude": 50.941, "longitude": -0.799, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United Kingdom", "flag": "\ud83c\uddec\ud83c\udde7"}, {"name": ["UK West"], "latitude": 53.427, "longitude": -3.084, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United Kingdom", "flag": "\ud83c\uddec\ud83c\udde7"}, {"name": ["France Central"], "latitude": 46.3772, "longitude": 2.373, "fillKey": "azure", "provider": "Microsoft Azure", "country": "France", "flag": "\ud83c\uddeb\ud83c\uddf7"}, {"name": ["South Africa North"], "latitude": -25.73134, "longitude": 28.21837, "fillKey": "azure", "provider": "Microsoft Azure", "country": "South Africa", "flag": "\ud83c\uddff\ud83c\udde6"}, {"name": ["Canada Central"], "latitude": 43.653, "longitude": -79.383, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Canada", "flag": "\ud83c\udde8\ud83c\udde6"}, {"name": ["Australia Central"], "latitude": -35.3075, "longitude": 149.1244, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Australia", "flag": "\ud83c\udde6\ud83c\uddfa"}, {"name": ["Central India"], "latitude": 18.5822, "longitude": 73.9197, "fillKey": "azure", "provider": "Microsoft Azure", "country": "India", "flag": "\ud83c\uddee\ud83c\uddf3"}, {"name": ["East Asia"], "latitude": 22.267, "longitude": 114.188, "fillKey": "azure", "provider": "Microsoft Azure", "country": "China", "flag": "\ud83c\udde8\ud83c\uddf3"}, {"name": ["Southeast Asia"], "latitude": 1.283, "longitude": 103.833, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Singapore", "flag": "\ud83c\uddf8\ud83c\uddec"}, {"name": ["Central US"], "latitude": 41.5908, "longitude": -93.6208, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["East US 2"], "latitude": 36.6681, "longitude": -78.3889, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["West US"], "latitude": 37.783, "longitude": -122.417, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["North Central US"], "latitude": 41.8819, "longitude": -87.6278, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["South Central US"], "latitude": 29.4167, "longitude": -98.5, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["Japan West"], "latitude": 34.6939, "longitude": 135.5022, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Japan", "flag": "\ud83c\uddef\ud83c\uddf5"}, {"name": ["Japan East"], "latitude": 35.68, "longitude": 139.77, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Japan", "flag": "\ud83c\uddef\ud83c\uddf5"}, {"name": ["Brazil South"], "latitude": -23.55, "longitude": -46.633, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Brazil", "flag": "\ud83c\udde7\ud83c\uddf7"}, {"name": ["Australia East"], "latitude": -33.86, "longitude": 151.2094, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Australia", "flag": "\ud83c\udde6\ud83c\uddfa"}, {"name": ["Australia Southeast"], "latitude": -37.8136, "longitude": 144.9631, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Australia", "flag": "\ud83c\udde6\ud83c\uddfa"}, {"name": ["South India"], "latitude": 12.9822, "longitude": 80.1636, "fillKey": "azure", "provider": "Microsoft Azure", "country": "India", "flag": "\ud83c\uddee\ud83c\uddf3"}, {"name": ["West India"], "latitude": 19.088, "longitude": 72.868, "fillKey": "azure", "provider": "Microsoft Azure", "country": "India", "flag": "\ud83c\uddee\ud83c\uddf3"}, {"name": ["Canada East"], "latitude": 46.817, "longitude": -71.217, "fillKey": "azure", "provider": "Microsoft Azure", "country": "Canada", "flag": "\ud83c\udde8\ud83c\udde6"}, {"name": ["West Central US"], "latitude": 40.89, "longitude": -110.234, "fillKey": "azure", "provider": "Microsoft Azure", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["Korea South"], "latitude": 35.1796, "longitude": 129.0756, "fillKey": "azure", "provider": "Microsoft Azure", "country": "South Korea", "flag": "\ud83c\uddf0\ud83c\uddf7"}, {"name": ["Korea Central"], "latitude": 37.5665, "longitude": 126.978, "fillKey": "azure", "provider": "Microsoft Azure", "country": "South Korea", "flag": "\ud83c\uddf0\ud83c\uddf7"}, {"name": ["ap-mumbai-1"], "latitude": 19.073, "longitude": 72.871, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "India", "flag": "\ud83c\uddee\ud83c\uddf3"}, {"name": ["ap-seoul-1"], "latitude": 37.566, "longitude": 126.977, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "South Korea", "flag": "\ud83c\uddf0\ud83c\uddf7"}, {"name": ["eu-frankfurt-1"], "latitude": 50.09, "longitude": 8.69, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "Germany", "flag": "\ud83c\udde9\ud83c\uddea"}, {"name": ["eu-zurich-1"], "latitude": 47.376, "longitude": 8.541, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "Switzerland", "flag": "\ud83c\udde8\ud83c\udded"}, {"name": ["uk-london-1"], "latitude": 51.51, "longitude": -0.129, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "United Kingdom", "flag": "\ud83c\uddec\ud83c\udde7"}, {"name": ["ca-toronto-1"], "latitude": 43.653, "longitude": -79.383, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "Canada", "flag": "\ud83c\udde8\ud83c\udde6"}, {"name": ["us-ashburn-1"], "latitude": 39.04, "longitude": -77.48, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["us-phoenix-1"], "latitude": 33.44, "longitude": -112.07, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["ap-sydney-1"], "latitude": -33.868, "longitude": 151.209, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "Australia", "flag": "\ud83c\udde6\ud83c\uddfa"}, {"name": ["ap-tokyo-1"], "latitude": 35.689, "longitude": 139.691, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "Japan", "flag": "\ud83c\uddef\ud83c\uddf5"}, {"name": ["sa-saopaulo-1"], "latitude": -23.553, "longitude": -46.631, "fillKey": "oracle", "provider": "Oracle Cloud", "country": "Brazil", "flag": "\ud83c\udde7\ud83c\uddf7"}, {"name": ["PAR 1"], "latitude": 48.85, "longitude": 2.29, "fillKey": "scaleway", "provider": "Scaleway", "country": "France", "flag": "\ud83c\uddeb\ud83c\uddf7"}, {"name": ["AMS 1"], "latitude": 52.37, "longitude": 4.9, "fillKey": "scaleway", "provider": "Scaleway", "country": "Netherlands", "flag": "\ud83c\uddf3\ud83c\uddf1"}, {"name": ["Newark"], "latitude": 37.44, "longitude": -95.58, "fillKey": "linode", "provider": "Linode", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["Europe"], "latitude": 48.856, "longitude": 2.352, "fillKey": "bso", "provider": "BSO Network", "country": "France", "flag": "\ud83c\uddeb\ud83c\uddf7"}, {"name": ["V\u00e9nissieux"], "latitude": 45.697, "longitude": 4.8855, "fillKey": "oci", "provider": "OCI Cloud", "country": "France", "flag": "\ud83c\uddeb\ud83c\uddf7"}, {"name": ["Asia Pacific (Mumbai)"], "latitude": 19.06, "longitude": 72.88, "fillKey": "aws", "provider": "Amazon Web Services", "country": "India", "flag": "\ud83c\uddee\ud83c\uddf3"}, {"name": ["Asia Pacific (Osaka)"], "latitude": 34.67, "longitude": 135.5, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Japan", "flag": "\ud83c\uddef\ud83c\uddf5"}, {"name": ["US East (Ohio)"], "latitude": 40.42, "longitude": -82.9, "fillKey": "aws", "provider": "Amazon Web Services", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["Asia Pacific (Sydney)"], "latitude": -33.87, "longitude": 151.21, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Australia", "flag": "\ud83c\udde6\ud83c\uddfa"}, {"name": ["Asia Pacific (Tokyo)"], "latitude": 35.69, "longitude": 139.69, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Japan", "flag": "\ud83c\uddef\ud83c\uddf5"}, {"name": ["South America (Sao Paulo)"], "latitude": -23.55, "longitude": -46.63, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Brazil", "flag": "\ud83c\udde7\ud83c\uddf7"}, {"name": ["Asia Pacific (Singapore)"], "latitude": 1.35, "longitude": 103.82, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Singapore", "flag": "\ud83c\uddf8\ud83c\uddec"}, {"name": ["Canada (Central)"], "latitude": 50.56, "longitude": -100.17, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Canada", "flag": "\ud83c\udde8\ud83c\udde6"}, {"name": ["Asia Pacific (Seoul)"], "latitude": 37.57, "longitude": 127.0, "fillKey": "aws", "provider": "Amazon Web Services", "country": "South Korea", "flag": "\ud83c\uddf0\ud83c\uddf7"}, {"name": ["US West (Oregon)"], "latitude": 41.64, "longitude": -83.47, "fillKey": "aws", "provider": "Amazon Web Services", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["US West (N. California)"], "latitude": 40.06, "longitude": -122.48, "fillKey": "aws", "provider": "Amazon Web Services", "country": "United States of America", "flag": "\ud83c\uddfa\ud83c\uddf8"}, {"name": ["EU (Frankfurt)"], "latitude": 50.11, "longitude": 8.68, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Germany", "flag": "\ud83c\udde9\ud83c\uddea"}, {"name": ["EU (Ireland)"], "latitude": 53.35, "longitude": -6.26, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Ireland", "flag": "\ud83c\uddee\ud83c\uddea"}, {"name": ["EU (London)"], "latitude": 51.51, "longitude": -0.13, "fillKey": "aws", "provider": "Amazon Web Services", "country": "United Kingdom", "flag": "\ud83c\uddec\ud83c\udde7"}, {"name": ["EU (Paris)"], "latitude": 48.85, "longitude": 2.35, "fillKey": "aws", "provider": "Amazon Web Services", "country": "France", "flag": "\ud83c\uddeb\ud83c\uddf7"}, {"name": ["EU (Stockholm)"], "latitude": 59.32, "longitude": 18.07, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Sweden", "flag": "\ud83c\uddf8\ud83c\uddea"}, {"name": ["Asia Pacific (Hong Kong)"], "latitude": 22.33, "longitude": 114.2, "fillKey": "aws", "provider": "Amazon Web Services", "country": "China", "flag": "\ud83c\udde8\ud83c\uddf3"}, {"name": ["Africa (Cape Town)"], "latitude": -33.924, "longitude": 18.424, "fillKey": "aws", "provider": "Amazon Web Services", "country": "South Africa", "flag": "\ud83c\uddff\ud83c\udde6"}, {"name": ["EU (Milan)"], "latitude": 45.464, "longitude": 9.189, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Italy", "flag": "\ud83c\uddee\ud83c\uddf9"}, {"name": ["Middle East (Bahrain)"], "latitude": 26.066, "longitude": 50.557, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Bahrain", "flag": "\ud83c\udde7\ud83c\udded"}, {"name": ["Asia Pacific (Jakarta)"], "latitude": -6.1753, "longitude": 106.82, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Indonesia", "flag": "\ud83c\uddee\ud83c\udde9"}, {"name": ["Middle East (UAE)"], "latitude": 24.0, "longitude": 54.0, "fillKey": "aws", "provider": "Amazon Web Services", "country": "United Arab Emirates", "flag": "\ud83c\udde6\ud83c\uddea"}, {"name": ["Asia Pacific (Hyderabad)"], "latitude": 17.36, "longitude": 78.474, "fillKey": "aws", "provider": "Amazon Web Services", "country": "India", "flag": "\ud83c\uddee\ud83c\uddf3"}, {"name": ["EU (Spain)"], "latitude": 40.416, "longitude": -3.703, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Spain", "flag": "\ud83c\uddea\ud83c\uddf8"}, {"name": ["EU (Zurich)"], "latitude": 47.374, "longitude": 8.541, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Switzerland", "flag": "\ud83c\udde8\ud83c\udded"}, {"name": ["Asia Pacific (Melbourne)"], "latitude": -37.814, "longitude": 144.9631, "fillKey": "aws", "provider": "Amazon Web Services", "country": "Australia", "flag": "\ud83c\udde6\ud83c\uddfa"}, {"name": ["GRA"], "latitude": 50.986, "longitude": 2.125, "fillKey": "ovh", "provider": "OVHcloud", "country": "France", "flag": "\ud83c\uddeb\ud83c\uddf7"}], "zoom": false, "geographyConfig": {"hideAntarctica": true, "hideHawaiiAndAlaska": false, "popupOnHover": true, "highlightOnHover": true}, "bubblesConfig": {"radius": 7, "borderWidth": 1, "borderColor": "#FFFFFF", "popupTemplateFormat": "<div class=\"hoverinfo\"><b style=\"color: ${data.color};\">${data.provider}</b><br>${data.name}</div>"}, "arcConfig": {"strokeColor": "#DD1C77", "strokeWidth": null, "arcSharpness": 1, "popupOnHover": false, "popupTemplateFormat": "<div class=\"hoverinfo\"><strong>${data.name}</strong></div>"}}
The measurements
Here's the goal
- We want to record the time to first bytes (TTFB)
- We don't want to record DNS time
- We don't want to record pre-flight requests
- We don't want to involves data cache
We use the browser's native XMLHttpRequest API to track performance of each request. Javascript allows to watch the following status
| State | Description | Cloud Mercato's usage |
|---|---|---|
| UNSET | Client has been created. open() not called yet. | - |
| OPENED | open() has been called. | Probe start period |
| HEADERS_RECEIVED | send() has been called, and headers and status are available. | Probe end period |
| LOADING | Downloading; responseText holds partial data. | - |
| DONE | The operation is complete. | - |
Each test starts by a dry request which:
- Make sure the endpoint is available
- Fill the DNS cache
- Process CORS
The results
You can make a live visualisation of tests here in the Observer or obtain a larger vision on our Price/Performance Portal.
Except IP for geo-localization of clients and User agent to get an idea of the device, we do not collect any personnal data.