Update #8: 10th November 2019
Sadly, it looks like there are still ongoing issues with the Falkland Islands Internet service after receiving several emails and social media messages yesterday about unreliable connections to some web sites. This is a classic symptom of the instability of Internet routing paths caused by connectivity dropouts or DDoS attacks.
Whatever the cause(s), it is yet again creating frustration in the islands user community.
Instability is shown in the Oracle Internet Intelligence Service below. This shows the times and dates things when things really deteriorated.
The same graph for St Helena shows how boring a BGP plot should look, but also indicates that the issues have nothing to do with satellite infrastructure shared with St Helena or Ascension Island.
Sure Falkland Islands have not issued a statement since November 4th when they stated: “…we are continuing to see intermittent network faults from our upstream service provider”.
They have not issued a statement stating that the issue has been finally cured, or issued any other update to their customers that I have seen about the seemingly on-going issues.
I’ll be in Stanley from the 14th November to the 3rd December, so f you would like to talk with me or meet up, just Facebook message me, email on firstname.lastname@example.org or phone me on my local phone +500 62971.
Update #7: 6th November 2019
Another day comes and goes, but this really looks to be a much better day than for a long time with access to the Falkland Islands DNS server from the UK being consistent for the last 24 hours. There have only been a couple 0% Pings throughout the period.
I would hope that Falkland Islanders are now seeing the Internet Quality of Experience returning to what it was before the satellite capacity reduction raised its head in October.
Not much more to say in this post which is pleasing.
Update #6: Bonfire night, 5th November 2019
Sure Falkland Islands sent out the following email to its customers yesterday afternoon:
Yet another Groundhog Day it seems, very bad early in the morning but also throughout the evening. In fact, the evening deterioration seems to be only a little better than the quota-free night period!
I believe this to be one of the most severe and protracted Internet deterioration of Internet performance in the last decade. Remember this issue started back in mid-October.
St Helena has some routing issues in October, but nothing on the scale that has been experienced in the Falkland Islands.
How serious the issue still is can be seen in the BGP plot below. The instability in BGP routes means that the Falkland Islands were probably disconnected from the Internet completely for short periods at these times – even yesterday.
An explanation from a Cisco paper says:
“… instability happens when there is internal congestion in some links within an AS and this causes the TCP connection between two BGP routers of that AS to time out. … This problem can recur several times and causes the routing tables of the routers in the AS Y to become unstable.”
I really would expect a factual explanation explaining the real cause of this issue to be put into the public domain and, more importantly, what will be done to prevent reoccurrence.
Could it be a Possible Distributed Denial of Service (DDoS) attack?
Google defines a Distributed Denial of Service attack as:
“A distributed denial-of-service (DDoS) is any type of attack where the attackers (hackers) attempt to prevent legitimate users from accessing the service. In a DoS attack, the attacker usually sends excessive messages asking the network or server to authenticate requests that have invalid return addresses.”
Indeed, it has been claimed in the past that certain deterioration of service incidents on the Falkland Islands were caused by DDoS attacks and Sure did say in their post above that “…the current issue is different to one prior to the 31st October though that is still ongoing”. The previous cause was reduced satellite capacity causing congestion.
The warning signs of a Denial of Service attack are similar to what the Falkland Islands are currently experiencing:
“Some symptoms of a DDoS attack include network slowdown, spotty connectivity on a company intranet, or intermittent website shutdowns. No network is perfect, but if a lack of performance seems to be prolonged or more severe than usual, the network likely is experiencing a DDoS and the company should take action.”
I have to say that this is hard to ascertain without inside knowledge, but I have to say it does not feel like a DoS attack. The main reason is that it has gone on for so long and has been so severe. Usually, once the source of the attack has been identified it can be blocked quite quickly. Why should a lowly ISP located in a remote part of the world be picked on as there is usually a financial cause of a DDoS attack? Is it too much of a coincidence to have a major DDos attack at the same time as having a satellite capacity issue?
Moreover, DDoS attacks have been around since the early days of the Internet and all ISPs would – or should – have the security mechanisms and mitigation processes in place to resolve incidents when they occur. I would certainly expect Sure to be competent in this area.
This all sounds like an exceedingly challenging issue so I would like to wish the best of luck to Sure’s engineers with getting this issue rectified.
Whatever the cause, I really do hope that this has cleared up before I arrive on the islands on the 15th November.
Update #5: 4th November 2019
We’re back! I am pleased to report that access to OpenFalklands has been restored since 3:10 p.m. yesterday as can be seen in the plot below. It has been inaccessible to Falkland Islanders since the 30th of October.
The plot below is the result of Ping monitoring one of the FK DNS server located on the Falkland Islands for the last 20 hours. Yet again, there seems to have been performance issues during Sunday morning when usage was high.
Though it looks to have settled down a later, there were still many comments about the service being unreliable and slow throughout the day and evening congestion can be seen in the Ping plots throughout the day. It clearly worsens during the night, which would be expected.
I can’t tell from the UK whether the Falklands Internet service has now been properly restored to what it was prior to the problems encountered throughout the latter part of October, but we will see.
Whatever the situation, it is clear that the Falkland Islands Internet service is ‘living on the edge’ and is highly stressed at times of peak usage
Update #4: 3rd November 2019
This is the result of pinging the .FK DND server for 24 hours starting early morning on the 2nd November.
As you can see on the left-hand side of the plot, yesterday did not start well but seems to have settled down around 13:00. The quota-free period on the right-hand side of the graph looks normal as well.
It’s even looking good from the Falkland Islands end with this ping plot of Facebook.com
I hope that this short update will be the last for this issue and that things have been sorted out once and for all.
Update #3: 2nd November 2019
The 24-hour ping plot on one of the Falkland Island’s DNS servers may be seen in the image below:
The wide red block on the left is the Sure Planned Outage starting around 07:00 on the morning of the 1st November intended to be used to correct the fault and restore satellite capacity back to ‘normal’.
After a period of the service being unsettled, access to the DNS server looks to have returned to normal with consistent performance through to midnight when the quota-free period started.
During this period congestion occurred but this was to be expected.
Oracle’s Squish service also shows the Planned Outage during the morning of the 1st November.
It would seem that the Falkland Islands broadband service has returned to normal at the moment and I can only hope this is the case. Only Falkland Islanders are able to know this as they have firsthand experience.
Late on the 1st November 2019, Sure Falkland Islands posted the following message saying that service had been restored:
I will carry on monitoring.
Update #2: 1st November 2019
It’s 07:00 on the 1st of November in the rainy UK. The first thing I did before making a cup of tea was to take a look to see whether the degradation of the Falklands islands Internet broadband service had been resolved overnight.
Unfortunately, looking at the ping plot of the Falkland Islands-located DNS server over the last 20 hours shows that it does not look like it has.
The DNS server is still inaccessible for 12% of the time and severely slowed down at other times. Of course, with the 00:00 to 06:00 quota-free time, this would be the case anyway – but not quite as bad as this.
Pinging the DNS server over the satellite from the UK
Oracle’s Squish service also shows the service degradation during the morning of the 31st October.
As a test, the FK DNS server is only hosted locally on the Falkland Islands, I asked a Sure Falkland Islands customer to ping test the DNS server locally to see what its performance was like for a local. The graph below shows consistent results with an average ping time of around 40mS.
Pinging the DNS server over the satellite from Stanley
This is what I would expect as it shows that the degradation of the DNS server performance from outside the islands is purely down to extreme congestion on the satellite link due to reduced satellite capacity caused by the fault.
This is why the DNS server needs to be mirrored in other parts of the world to meet IANA’s rules for DNS hosting. It is expected that this will be sorted out in December.
Though it seems a bit daft showing these two graphs as Falkland Internet users are suffering the results of the satellite capacity problem first hand, here are graphs of ping monitoring www.googl.com for 24 hours undertaken by a Falklands Islands’ Internet user. It shows packet losses of up to 90% at times.
Pinging Google.com from the Falkland Islands
This must be the most prolonged period of severe broadband service degradation for many years. Two questions I’d like an answer to are:
The service outage has major consequences for the islands’ businesses and this is typical of a number of emails and messages I have received. Wow.
“…it was so slow at work today that we couldn’t even access XXXXX so we couldn’t sell anything on it as the link was not strong enough to even open let alone make a transaction, it really isn’t funny but there is nothing we can do about it.”
Has the deterioration been caused by Sure attempting to upgrade their equipment ready for the doubling of satellite capacity in December?
If it’s down to just equipment failure in Stanley or the UK, why is it taking so long to rectify?
Last night, Sure Falkland Islands posted the following message.
I will carry on monitoring through the day.
Update #1: 31st October 2019
Halloween Falkland Islands Internet performance
I was notified at 06:00 this morning (31st October 2019) by a Sure Falkland islands customer that there were ongoing performance issues with the islands’ Internet service this morning. I immediately started monitoring several servers located on the islands to see what was happening.
At around noon local times Sure Falkland Islands posted the following message on their FaceBook page explaining the issue. Unfortunately, I am unable to determine the exact cause.
With on-going issues surrounding the location and lack of diversity of the FK DNS equipment, I monitored some critical Falkland Islands servers from the UK – one of the DNS servers 220.127.116.11 and horizon.co.uk server associated with the islands’ Horizon email service.
Whatever the technical cause of the issue; whether it caused by Sure-managed equipment, the Intelsat satellite or Vodafone, (Sure’s Internet network supplier provider in the UK) it has gone on for most of the day by the looks of it.
The technical issue has significantly reduced the capacity of the satellite link(s), which has caused a very high level of traffic congestion at times of peak usage. Indeed, early this morning it looks like the service pretty much collapsed with many periods when connectivity failed which was repeated at lunchtime.
It feels like a continuation of the problems experienced a few days ago.
The plots below show how the two Falkland Islands servers have responded to ping testing between 08:00 and 14:00 local time today.
It can be clearly seen that the two servers experienced nearly identical degraded performance.
When there is no satellite congestion, the black line should be a simple straight line showing the 580mS satellite round-trip delay. Black spikes show that congestion is significantly slowing down ping responses and the red lines show that the Falkland Islands servers were unreachable.
Objectively, between 10 and 14% of packet loss was experienced from the UK.
During the red and the black spike periods, the FK DNS server was unreachable and web browsing and emailing becoming unreliable, slow and frustrating.
I’ll be posting further data later so I will be able to see the transition to a restored.
27th October 2019
Unfortunately, I only realised that there had been an issue with the performance of the Falkland Islands’ broadband services a little late, so the data I have collected is rather thin. However, this is what I have gleaned.
Following complaints on Facebook, Sure Falkland Islands posted the following on their Facebook page.
However, a closer inspection showed that the issue was significant in the Falkland Islands. I do not understand the statement that “Amazon, Google as well as Facebook were having worldwide issues”. This statement is probably incorrect, as all these applications run on their own distributed resilient servers and mostly their private international networks so it would be virtually impossible for all to be having problems at the same time.
The commonality between these applications is the single point of failure – the Falkland Islands satellite link and Sure Falkland Islands infrastructure managing it.
Oracle Internet intelligence
As I was unaware of the issue until the 26th October 2019, I do not have any data showing broadband performance at the time, but a quick look at Oracle Internet Intelligence tells the actual story. Oracle flagged a major issue with Internet connectivity to the Falkland Islands as can be seen below.
Oracle Internet Intelligence Internet fault map
Unlike the major outage experienced in late April 2019, this only affected the Falkland Islands. The Falkland Islands outage last April was a knock-on effect caused by an issue in St Helena. This cause is not the case this time.
Oracle’s historical Traceroute graph shows a major outage to the Falkland Islands that lasted for over 6 hours in the early hours of 25th October 2019.
The following explains what the graph means.
“As part of our Internet measurement infrastructure, we run millions of traceroutes per day to all parts of the Internet from hundreds of globally distributed measurement servers. In this panel, we highlight instances where the number of observed responding traceroutes drops below a historical baseline.”
Another Oracle graph shows the performance of the FK Domain name Server (DNS) which also suffered the outage.
.FK Domain Name Server performance
This graph shows that the FK DNS servers were inaccessible from the worldwide Internet, meaning that FK web sites and FK email were affected. The negative aspects of the lack of FK resilience have been well discussed in my post. I do hope that this will be corrected in December 2019. Responsive DNS performance lies at the heart of the Internet functionality, and nothing should be allowed to interfere with that.
The final graph shows that the problem with the Falkland Islands Internet had been building up for quite a few days.
Border Gateway Protocol (BGP) Performance
The following explains what the graph means.
“Border Gateway Protocol (BGP) is the mechanism by which entities such as Internet Service Providers (ISPs), commercial enterprises, and universities exchange information on how to reach ranges of Internet Protocol (IP) addresses (prefixes).
The lack of stability in BGP routing would mean that, from a user perspective, the Internet would be unreliable and have unpredictable performance. This routing issue would explain why Falkland Islands Internet users were having problems, not that “Amazon, Google, as well as Facebook, were having worldwide issues”.
I do not know why the performance of BGP routing has not resumed since the outage on the 25th October 2019 – does this indicate continuing problems?
26th October Performance monitoring.
Although I only learnt of the issue on the 26th October, I thought it was worthwhile monitoring the performance of the FK DNS server from the UK. This is the result for monitoring over the evening of the 26th and the morning of the 27th October 2019.
Pinging the Falkland Islands based DNS server 18.104.22.168
Y-scale is 2.6 seconds!
The left-hand part of the graph shows the performance of the Falkland Islands-based DNS server during the evening of the 26th October. Responses to ‘pings’ were generally consistent. This reflects the round time trip delay of 580mS of the satellite link. This part of the graph indicates that IP traffic is being effectively managed by Sure (slowed down) to ensure that consistent performance is experienced by all users. THis activity is otherwise known as implementing a ‘Fair Usage Policy’.
It could be construed from this that the performance issue that started earlier in the month has been resolved.
However, the right-hand part of the graph shows what happened to DNS requests during the quota-free period that started at midnight. It shows that though the Internet access service is still running the performance is abysmal with DNS responses averaging 1.5 seconds with many dropouts.
There has been a lot of talk of working towards a quota-free Internet access service, but what I take from this is that this is an unachievable ambition when using shared satellite capacity to access the Internet.
I hope that this throws a little more light on the frustration that has been shown on Facebook in recent days about the poor performance of the Falkland islands Internet service.
Copyright: October 2019, OpenFalklands