May 14th update
Saint Helena and the Falkland Islands Face Simultaneous Outages. (14-05-19)”]
On 30 April 2019, internet service in the Falkland Islands and Saint Helena suffered simultaneous outages.
We often see large outages in islands or remote areas due to limited sources for internet connectivity and/or extreme weather events.
In this case, Sure South Atlantic Ltd is only source of internet connectivity for these two British Overseas Territories. A technical glitch during Sure’s scheduled maintenance extended what was supposed to be an hour of planned downtime into a 4-hour blackout. In his blog, Chris Gare, a former telecoms consultant to the Falklands Government, highlighted another fallout of the blackout in the South Atlantic, namely that the DNS zone for the .fk TLD (which is also run by Sure South Atlantic Ltd) went completely down. It turns out that Sure hosts both NS servers for .fk (ns1.horizon.net.fk = 195.248.193.250, ns2.horizon.net.fk = 195.248.193.251) in the same routed prefix essentially negating the redundancy of multiple NS servers.
The outage also highlights the efforts of Saint Helena to secure a source of high-speed internet other than satellite, which can be expensive, high-latency and capacity constrained. For several years, the “Move This Cable” campaign has advocated for a modification of the route of the proposed SAEx submarine cable. According to its website, “there are over 4,200 people living on one of the most isolated islands in the world” and that landing the SAEx cable in Saint Helena would allow the locals to “finally join the information society, … improve standards of education and healthcare, as well as offering new economic prospects.” After securing a grant of € 21.5 million for the branch from the European Union, funding of the trunk cable across the South Atlantic has become uncertain recently.
Source: Oracle Internet Intelligence
April 15th update
Intelsat-29e satellite suffers fuel leak, spotted drifting along GEO arc. Intelsat’s first Epic-series high-throughput satellite, Intelsat-29e, is drifting in orbit after back-to-back anomalies, forcing the company to shift customers to other spacecraft. (10-04-19).
Take a look at the News page for the full report. Is this the cause of the Internet ‘brown-out’?
I’ve written a lot in previous posts about the causes of satellite capacity congestion, how satellite congestion is managed though the use of quotas and ‘traffic shaping’ and how increased Contention Ratios result in congestion and inevitably slowing the Falkland Islands’ broadband service down.
As I’ve explained in previous posts, one of the principle metrics used to measure contention is a ‘Ping test’ and I have used this to take an objective look at the Falkland Islands’ broadband service on April 2nd/3rd 2019.
April 14th 2019 update
The Ping test undertook on the 2nd/3rd April 2019 coincided with an announced equipment fault located at the Intelsat uplink site at Whitehill, Oxfordshire in the UK. It is unknown whether this fault occured in Sure SA managed equipment or in the edge router of their upstream Internet backbone provider, Vodafone (Vodafone bought C&W’s UK assets a while back).
On the 11/12th of April I ran a further 24-hour Ping test to see whether things had improved after Sure SA announced that the fault had been corrected on the 8th April 2019.
The results may be seen below when plotted on the same vertical scale as the graph shown above. The vertical scale shows delays of up to 1000mS. I’m pleased to be able to report that Ping delays on the 11/12th April have dropped significantly showing that the fault has been corrected.
Ping test results for the 11/12th April 2019 with a 1000mS vertical scale.
I replotted the Ping test data using a 200mS vertical scale to show Ping delays in more detail. It may be observed that there are still quite a few large Ping delays spread fairly evenly throughout the day indictating congestion. Although this level of congestion would not be acceptable in the UK, that is not a fair comparison to make with the Falkland Islands. A 210Mbit/s satellite link shared by 1800+ Sure SA Internet customers, it would be expected that there will be congestion through much of the day.
Again I’d like to retiterate that the Ping test only indicates that congestion exists and not how slow the Internet will be at any time.
Ping test results for the 11/12th April 2019 with a 200mS vertical scale.
Sure Speedtest
During the period when Sure had a UK equipment fault the following image was posted on Facebook. What is interesting about this image is that it graphically shows the differance between downloading a file transiting the satellite while simultaneously running Sure SA’s speedtest from a server located in Stanley.
An anecdotal comment made at the time said it took 15 minutes to download a 3MB file (an edition of Penguin News I believe).
The local Speedtest server is only useful for checking that the Internet package you are paying for is actually set up correctly by Sure or to check that your home Wi-Fi is behaving itself.
8th April 2019 update
Sure sent out the foillowing email to its customers on the 5th April:
Date: 5 April 2019 at 16:20:25 GMT-3
Subject: Broadband Disruption – UpdateWe wish to advise customers that internet services have been restored to full capacity now during peak times, just in time for the weekend. The Sure IP Engineering team have worked tirelessly with our service providers to resolve this Broadband Disruption which was due to an equipment failure in the UK.
We apologise for any inconvenience that this may of caused and will be giving all Broadband customers an extra 10% of data on top of your usual monthly allowance for the month of April and will be available to use from Monday.
What is a Ping test?
An appropriate analogy is transporting a big lorry load of bricks (Internet packets) across the country. The bandwidth is high (i.e. the number of bricks), but the latency or delay is huge (i.e. the time it takes to cross the country). The Ping test measures how long it takes the lorry to cross the country but not how big the load the lorry is carrying. Ping times do not change with bandwidth.
However, if a Ping ‘request’ or ‘response’ gets held up and queued behind a lot of other data packets then it will be significantly delayed, reflecting that the broadband service is slowing down. This is what happens when there is congestion at peak times of usage.
A PIng test run on my PC showing a Ping of 10mS to the test server.
For the technically minded, Ping tests use the Internet Control Message Protocol (ICMP) which is a protocol that can test whether a remote router, switch or server is accessible. If that router, or server is overloaded through excessive demand its IP traffic is held in a queue leading to the router or server not responding to a Ping Request instantly.
Internet standard RFC 1122 mandates hosts to always send a reply to a Ping request but this is not always the case as ICMP can be seen as a security risk, A ‘flood’ ping option exists that is able to sending requests as fast as possible in an attempt to determine the response of the network under high-load conditions. Of course, I have not used that!
Ping testing the Falkland Islands’ broadband service.
In the absence of any formal Quality of Experience monitoring of the Falkland Islands’ broadband service at present, I thought it might be interesting to see how the service responds to a Ping test running for 24-hours. This will not provide any data about download or upload speeds, but it will indicate when there is high congestion on the Falkland Islands’ satellite link.
I used a program called PingPlotter (which can be used for 14 days for free). The maximum length of ICMP messages is 576 bytes so the total amount of Mb used in the test was (24x60x2)x576x2 = 3.3mByte which is a nominal amount really.
PingPlotter plots a ‘trace’ route from a web browser to a web server that is specified. In this case, from my home PC to the server in the Falkland Islands. The end-to-end path Ping request and their responses take in the test can be seen below and includes the selected web server, Sure’s Falkland Islands network, the satellite, Sure’s equipment in the UK, the UK Internet and my PC.
The path from the Falkland Islands’ server to the PC in the UK
You can see from the ‘traceroute’ below that packets from my PC pass through 15 routers before they arrive at the destination server in the Falkland Islands. The first 10 hops are in the UK, the 11th hop is the last router in the UK which is located just before the satellite up-link. The final hops are in the Falkland Islands with hop 15 being the server. I have blanked out the UK servers for privacy reasons. Note the big delay after the router 62.208.72.82 when the packets transit the satellite.
It takes 15 hops to go from the UK to the Falkland Islands server.
There is an approximate 530mS delay caused by the geo-stationary satellite which affects EVERY Internet Protocol packet that passes over it. In essence this is a fixed delay.
Removing UK Internet from the Ping data.
For this test I sent a Ping request every minute for 24 hours on April 2nd/3rd 2019.
As I only wanted to look at the Ping data that included the server, the Falkland Islands broadband network and the satellite itself, I needed to exclude the UK path from the satellite downlink in Whitehill to my personal computer i.e. in the picture above, this is the first 11 hops.
There is always a small amount of congestion on UK Internet backbones, so I subtracted the occasional UK delays from the end-to-end delays to ensure accuracy. The delays left were clearly delays on the local Falkland Islands network and the satellite link. I achieved this by also pinging the Falkland Islands satellite downlink server – 62.208.72.82 for 24-hours as well.
Also, as every Ping request and responce suffers a consistent 530mS delay or latency, I subtracted this time from the overall delay or latency as measured by PingPlotter.
Ideally, there should be little variation during a 24-hour period as this would indicate very low congestion, however this will not be the case for the Falkland Islands’ broadband service as everyone is aware there is congestion at certain times of the day which slows the service down.
Ping test results.
After taking out a small number of poor Pings on the UK section of the path to the Falkland Islands’ server and subtracting the fixed satellite link latency, I plotted the results on an Excel graph which may be seen below:
Ping test results for a 24-hour period starting at 11:00 FKST on 2nd/3rd April 2019. (No-quota time is shown in pink from 00:00 to 06:00)
So how do we interpret the results?
Caveat: The graph showing Ping delays or latency is only an indicator of congestion and not what a user’s experience might be as far as service slow-down is concerned. They do provide a trustworthy indication of congestion though.
The horizontal axis is Falkland Islands time and the vertical axis is the delay or latency from the Stanley server to the far side of the satellite link while excluding the rest of the Internet path.
11:00 to 16:00 FKST
During this period consistently low Ping scores of around, of say, 15mS. This shows that there is little congestion on the broadband service in this period.
16:00 to 00:00 FKST
Starting around 16:00, congestion builds up rapidly to a peak at 17:30 and remains consistent through to late evening. Many of the Pings show a delay in excess of 300mS and if the satellite delay is added in this means a total delay of 830mS. I would consider this is a very high level of congestion and is half way to the level of service degradation experienced during the 00:00 to 06:00 quota free period. This would certainly be noticed by users of the broadband service.
00:00 to 06:00 FKST
This is the quota-free period when many download very large PC, mobile phone updates and peer-to-peer films. The link is very congested. The peak latency during this period is 900mS! Add in the satellite latency of 530mS and you get a total a 1.43 seconds delay. I would imagine it would be impossible to use any interactive Internet service during these times and even download protocols such as TCP/IP could be affected making it not so much unreliable as making downloads of large files exceeding slow due to error correction and re-send requests.
TCP/IP uses a technique called ‘slow-start’. Slow-start gradually increases the amount of data transmitted until it finds the network’s maximum carrying capacity so under high congestion it will never speed up.
As an aside, this is one of the reasons why no bandwidth or Falkland Islands’ satellite capacity will ever enough or optimal as protocols will gobble up every free Mbit/s that is available. Expecially TCP/IP.
In extreme congestion TCP/IP file download will not speed up (Source: Stackpath)
06:00 to 11:00 FKST
Congestion drops back down to almost non-existence except for a brief period of congestion.
The 24-hour clock used by Actual Experience coincides with the results of the Ping test.
The cause of the congestion detected by the Ping tests
Sure sent out an alert to its customer base at 16:27 on the 25th March 2019 alerting them to an issue with their UK equipment.
From: Horizon Postmaster
Sent: 25 March 2019 16:27
Subject: Broadband DisruptionWe wish to advise customers that they may experience some disruption to Internet services during peak times due an equipment failure in the UK, we are working hard to resolve this issue and hope to have service back up to full capacity as soon as possible. We apologise for any inconvenience that this may cause.
With a follow-up on April 2nd.
From: Horizon Postmaster
Sent: 02 April 2019 15:09
Subject: Broadband Disruption – UpdateWe wish to advise customers that they may still be experiencing some disruption to internet services during peak times due an equipment failure in the UK, we are working hard to resolve this issue and hope to have service back up to full capacity as soon as possible. We apologise for any inconvenience that this may cause.
Ping test thoughts
I had not known about the equipment faults before I undertook the Ping test, though through serendipity this is quite useful as the results show an almost worse-case scenario in relation to degradation of the Falkland Islands’ broadband service.
It sounds like a serious situation as it has been going on for so long. I will re-run the Ping tests when the faults have been corrected which I hope will be soon and that they will show an improved situation.
Chris Gare April 2019 Copyright: OpenFalklands
I use Smokeping. It’s free and can be installed onto a rasberry pi.
Excellent initiative. Well done!