We need to get a feeling for the quality and availability of the time data. It’s important to know how close each of our server estimates are to the UTC standard, and how quickly the servers are responding to requests for time (and if they ever fail to respond!).

This is a natural point in the article to highlight the amazing work done by the NTP Pool.

The NTP Pool project is an effort launched by Adrian von Bidder in 2003 that has consistently been the largest, most reliable cluster of network time servers in the world. The NTP Pool delivers time to hundreds of millions of digital clocks around the world.

It’s important to note that the Pool is 100% free. Period. Nobody pays to use it, and Ask Bjørn Hansen, the Pool’s maintainer since 2005, pays every dollar of all hosting costs out of his own pocket.

The NTP Pool includes a monitoring system to check on any submitted server. If that system indicates that a submitted server is delivering high quality time reliably, it will direct queries to the submitted server. If a participating server drifts wildly from UTC standard or is not responding to queries reliably, the Pool will remove the submitted server from service until it becomes “safe” to use again.

Public Time proudly submits all servers we deploy for inclusion in the NTP Pool—adding the NTP Pool’s reliability data as another source of monitoring for our deployed servers.

Public Time quickly noticed that the Pool’s monitoring data often differed from what we collected through our own monitoring systems. We found the NTP Pool monitoring system often marked our servers as far less reliable than our own data indicated.

While the research effort is still very much ongoing, Public Time has started to be persuaded that NTP monitoring is often a function of where. By that, we mean where a server is monitored from matters—the more “wheres” you use, the better!

At the time we started looking harder at the NTP Pool’s data in 2017, the NTP Pool monitoring system consisted of a single server in Los Angeles, California.

It appeared that the further that a server was from Los Angeles (more a metric in terms of milliseconds of latency than physical miles—though the two are related), the more likely the Pool rated the NTP server as “low reliability”.

The Public Time team started digging into some of the more interesting cases. One such example was the Public Time server in São Paulo, Brazil. The Brazil server looked terribly unreliable from the NTP Pool monitor in Los Angeles, ~10,000km/6,000 mi away.

Our monitoring servers showed huge variation:

• New York, ~7,500km/4,800 mi away, showed the São Paulo server to be just as unreliable as the Pool did.
• Frankfurt, Germany, 9,700 km/6,000 mi away, showed the Brazil server to be impressively accurate and reliable, though it did occasionally show a dropped response.
• Singapore, 16,000 km/10,000 mi away, showed the Brazil server to be both incredibly accurate and incredibly reliable, with well over 99% of requests getting responses.

Public Time is coming to the opinion that “positive” responses (i.e. the server is both accurately tracking to the UTC standard and reliably responding to NTP queries) can be given a fair bit of weight/relied upon. The issue is the same cannot be said for “negative” responses.

There are a LOT of factors outside the control of an NTP server operator. One of the biggest is the networks along the path from a computer making an NTP request to the target NTP server. If NTP queries fail to even make it to the target server, it’s not accurate to “score” the NTP server as unreliable as it never got the chance to reply! And vice versa if a reply successfully sent by the NTP server fails to make it back to the NTP client.

As such, the Public Time’s monitoring system algorithm has started taking a healthy bias approach, to try and accurately represent the health of a monitored NTP server:

Minority Positive: if 20-30% of our monitoring stations around the world are getting indications of good accuracy/reliability from a server, it’s safe to assume the server itself is functioning fine and thus give the server a score of “healthy.”

Majority Negative: if 80% or more of the global monitoring locations report poor accuracy or a large percentage of requests being dropped, it’s safe to assume a server is performing poorly.

Public Time’s effort to improve our ability to ensure high-quality time data from our global fleet of servers is always ongoing, but we wanted to give readers a snapshot in time of where we are as of today. Tomorrow—who knows??? 🙂

The post Global NTP Server Monitoring appeared first on Public Time.

Since NTP and Global Navigation Satellite Systems (GNSS) matured and computer connections have enabled global public access to these systems, it has become clearer by the decade how important it is to keep track of time.

Public Time sees global clock synchronization as an important effort to contribute to. The challenge of meeting that goal comes in many flavors — infrastructure, server upkeep, and public access are among the most concerning. One particular challenge that many — definitely including Public Time! — seem to struggle with is responding to one simple, fundamental question:

Why?

Anyone involved in the network time community, from those who develop software to help synchronize digital clocks or those who run the servers distributing highly-accurate time dreads this question: “Okay, I understand what you’re doing — but why are you doing it?”

It’s a topic with multiple layers, but at some level, the person asking the question is asking you to sell them. They want to hear your pitch to find out if it’s a bigger deal than they currently understand it to be, and if so, it may motivate them to get involved in some manner.

In my personal experience so far, it seems like many (most?) of the responses immediately run for “doom-and-gloom” scenarios:

“Without carefully-synchronized clocks, you can’t secure your network!”

“Without carefully-synchronized clocks, bank transactions will fail!”

“Without carefully-synchronized clocks, the power grids will fail!”

“Without carefully-synchronized clocks, THE OCEANS WILL BOIL.”

Okay, well, maybe not that last one — at least not due to lack of synchronized digital clocks. 🙂

Jokes aside, a lot of the justifications I’ve encountered when trying to clarify the Public Time stance on why we put so much time, effort, and money into synchronizing digital clocks is intensely negative. I’m reluctant to use the word “fearmongering,” but a lot of the public justifications used by the network time community aren’t terribly far removed from that.

There are times Public Time, and myself in particular, have been flirting with line between persuasive logic and fear-mongering — I’ll be the first to say that Public Time is still struggling with how to positively frame the discussion about why our mission is important. That said, history has shown that trying to sway opinions to your side by telling your audience that they’d better start seeing the world the way you do or else is not a terribly effective way to change hearts and minds. Beyond that, I’d like to see Public Time be one of the leaders of the charge, helping show there are more effective ways to generate interest (and donations!) in our cause that don’t rely on fear-based justifications for the importance of synchronized time.

The Public Time board has thought hard for a good while on how we’d like to talk about this topic. Our company’s position is still in its infancy, but one of the more promising threads we’re pulling on is passion.

The four founders of Public Time didn’t form Public Time because we were convinced that the internet and/or civilization would collapse if we didn’t start the company. It started out as a passion driven by the fact that there’s a “right answer” (meaning an international time standard, UTC) and due to our professional training, we have the ability to help computers around the world move towards that right answer. The four of us believe accurately adhering to an international standard is good on its own merits. We love the concept of timekeeping, we love watching seconds tick by, and we innately love the idea that we can help computers around the world start ticking in close beat to each other.

The four of us feel that there may be some parallels to network time community as a whole. It’s about passion first. Nothing brings a smile to my face quite like telling people that one of my most time-consuming hobbies is “distributing atomic time on a global scale.” Most people give me confused looks at first, but once I explain it, I find that my enthusiasm about the subject can be contagious, and people who were unaware of the issues, or were aware of but ambivalent to the issues, suddenly respond to my raw enthusiasm and bone-deep conviction that this all matters with enthusiasm of their own.

It may sound corny but, at a pretty fundamental level, humans are explorers. Few things stifle the urge to explore as well as fear. Unfortunately, fear has been the time community’s go-to sales pitch for too long. Accurate measurement of time is both valuable and fascinating. It’s integral to our daily lives yet it’s completely arbitrary, all the units and how to measure them made up entirely by humans.

Humans have agreed on a way to handle reckoning of time. Public Time will happily deliver our best estimate of the international time standard to absolutely any computer who wants it as a public good — meaning for no cost and with no access restrictions. That’s the Public Time mission statement in a nutshell.

Public Time can’t do it by ourselves, though. We need your help. Not only your help, your passion.

We don’t want you to offer your time, expertise, or money as a direct result of Public Time telling you the world’s oceans could catch on fire any moment now if we don’t stay on top of things. We want your help because we love working on teams with passionate believers that love tinkering with time.

Join us, won’t you?

The post The “Why” Question, the Network Time Community, and You! appeared first on Public Time.

Our “maintenance process” looked something like this: “Hey, I guess it’s been a few weeks and I need something to do during my lunch break. I should log into all the servers to make sure they’re up. While I’m there, I’ll apply the patches that have queued up.”

We were moving quickly enough that we spent more time planning future deployments than providing proper care and feeding to our small fleet of existing servers.

A Kurt Vonnegut quote unfortunately nails Public Time’s early philosophy towards server ops:

“Another flaw in the human character is that everybody wants to build and nobody wants to do maintenance.”

There was far more emotional payoff launching cloud servers in remote parts of the world and building relationships with other non-profits in our space. We knew maintenance was necessary, but the boring “ops” tasks always got put on the back burner — until they suddenly became urgent (like when a friend reported a server of ours was completely offline).

Skip forward to the present, where Public Time has 30+ servers across the globe. That laissez-faire approach to server ops doesn’t stand up well. It no longer was “a couple minutes to patch the fleet” manually — it was quickly starting to take the better part of an hour.

As we grew, like we hoped it would, the “when I am bored and need a distraction” approach to fleet ops wasn’t holding up.

Luckily there are plenty of software options out there for making sure our fleet of servers keep ticking. We knew we needed to prioritize immediate alerts with rapid response times for server fixes.

Since all our servers are Ubuntu-based, Landscape from Canonical seemed our best bet. With ways of easily deploying our base cloud servers, monitors that track server resources utilization, alerts if something’s down, and the ability to apply all patches nightly is proving to be a fantastic solution for us.

With Landscape helping us manage our servers, not only are we able to provide our services more reliably, but we can use this data to do better capacity planning as the fleet continues to rapidly grow.

The post Public Time’s Fleet Servers and Maintenance appeared first on Public Time.

And WOW, did we get some work done.

First off, Scott and I slated out an entire afternoon to tackle a problem we’ve wanted to fix for a while. Maintaining our servers in the beginning proved to be an easy ‘as-needed’ job. Until we started getting into the dozens of deployments realized that our lives in meatspace were distraction enough to miss the fact that several of our servers could stop responding without our knowing.

In the span of an afternoon, we implemented Ubuntu Landscape on every stratum 2 servers in the fleet. Landscape is a wonderful tool that allows us to check and patch our entire fleet at once, as well as optionally have it automatically apply security patches on a periodic basis (we went with nightly).

Our second objective was our baby: the Time Server App. We announced it months ago but this is when it all kicked off. I’ve loved time servers longer than I can remember but the thought of having an app the public could download always seemed like a fever dream before I met Cody, Eric, Tod, and Scott.

Months of work later, we rallied our resources and posted it to the Play store. The app was a fun challenge but we hadn’t drafted a solid Play Store pitch.

Eric and Cody both stepped up to the puzzle. Eric figured out all the hosting and submission details (for his first time doing so, he nailed it). And for the intro, Cody put together this:

Now if that music style sounds familiar, I imagine you’ve listened to Moby! That’s right, our little non-profit reached out and got permission from the artist to use his music to promote our app.

And we couldn’t have posted it at all without Tod’s rigorous QA on the app prior to launch.

Looking back, it’s surprising how our plans shift but our objectives remain the same. The app was by no means perfect at launch it’s being refined with regular patches. Our servers are operating more efficiently than ever and we can react to an issue with the software immediately.

Time is what we have and our time in Salt Lake was well spent.

The post Public Time JAM appeared first on Public Time.

Microsoft, Amazon, and Oracle have all recently announced plans of cloud deployments and data centers across South Africa and the Middle East. Public Time is excited to see them succeed as it will provide a foothold for our own efforts in the future.

South African Deployments

Back in 2017, Microsoft Azure announced two datacenters for cloud hosting in Cape Town and Johannesburg to be open for business in 2018. Despite missing their date, it’s a credit to Azure that this is the first time in their history that they’ve missed a go-live date. Both regions went live in early March 2019.

Hot on Microsoft’s heels, Amazon has plans to establish a cloud region consisting of three Availability Zones in 2020.

Public Time deployed to Azure in Johannesburg in May of 2019 and has been very pleased with our first NTP server deployment on Azure.

Middle East Deployments

Both Microsoft Azure and AWS have announced new cloud regions in the Middle East — United Arab Emirates and Bahrain respectively. Azure was first to open in June of 2019. AWS Bahrain became a month later.

Sometimes digital development is a game of inches and other times, it leaps miles. Public Time is delighted watching the cloud competition heat up in some of the most underserved markets in the world!

The post Groundbreaking: Africa and the Middle East! appeared first on Public Time.

For this meeting, we bravely took on the challenge of attempting a meeting in Georgetown (those who know the area will get that joke).

Happy to report that we’ve seen great strides in our hardware, software, and the community since our last meetup!

One of our major talking points was the Time Server App. Since its launch, we’ve been iterating on it, fixing bugs, and refining the quality of life aspects to better serve those who download it. The server aspect has been a riot of a puzzle considering how each model of a phone has separate protocols when it comes to serving up data — even something as simple as time. We look forward to continue cracking that code and getting the community in on the action with our open-source access.

Yes, Scott and Dan came to the meeting wearing the exact same shirt. No, they didn’t plan that. Or so they claim.

Something Brad has been working on in the last several months is our growing concern with the costs of our physical deployments. He’s been aggressively looking at improving the cost efficiency of our Stratum 1 solution, mostly cheaper alternatives to our first iteration of the hardware. We discussed our Meinberg card and server setup, which is an excellent rig but if we’re wanting to deploy dozens of these, our server cost is going to jump to the hundreds of thousands of dollars.

For a fledgling non-profit, we aren’t in a position where that’s realistic for us, so that’s why we’re looking into smaller-scale hardware for larger-scale deployment. Our various options include other open source cards. One interesting vein of research is using a raspberry pi as a host for a time serving framework. It would have to be a homebrew system — which is why we’re looking to our community for help!

Terry updated the team on our stratum one deployment slated for this year in Miami. We have received all the equipment we ordered for the deployment (a new server, and a bunch of spare coaxial cable), as well as identified the hosting provider we plan to use. At this point, we’re just working final logistics with the hosting provider.

After the meeting the team met up at Filomena, a Georgetown institution, for a fantastic Italian dinner and a wonderful chance to catch up and get some good laughs.

The post PublicNTP, Inc. July Board Meeting appeared first on Public Time.

A month later, on December 23rd, SpaceX successfully launched the Air Force’s first third generation GPS satellite, also known as “GPS IIIA-01” on top of a Falcon 9 rocket. While unquestionably a remarkable feat of engineering, this mission succeeded against a different set of challenging hurdles—twenty years of bureaucracy.

While the first GPS satellite was launched in 1978, the current Global Position System (GPS) did not reach fully operational status until 1995. While the first two GPS satellite generations served their purpose superbly for the time (compared to the previous services available), the always-marching advance of technology had progressed far beyond their estimations even then.

And 1995 hasn’t even gotten us to when mass-market cell phones or high-speed internet became widely-available!

In 1998, the Clinton administration recognized that a new generation of GPS satellites would be needed to keep up with the new 90’s advances. The GPS III initiative has been enthusiastically supported by every administration since its approval.

Multi-billion dollar contracts were written to get the system built—from Lockheed Martin’s major contributions in building the satellites, to Raytheon building the ground control system.

And some contracts changed hands. United Launch Alliance (ULA) received the launch contracts in 2012 for a launch in 2014. But due to delays in build deliveries and hiccups with late tech, the launch was postponed until 2018. In that four year span, the Air Force had sold the remaining launch contracts to SpaceX, and a later rocket-swap in 2017 handed ULA’s initial launch contract to SpaceX as well.

This switcheroo, by all optimistic estimates, would not delay the launch further. After a spring scare with some technical issues, the Falcon 9 heavy rocket launched the first of ten GPS IIIA satellites—with twenty-two IIIF satellites being commissioned last September (“F” to indicate a “follow-on” series).

We’re hopeful that GPS III is finally hitting its “launch stride”. The second GPS III space vehicle, GPS IIIA-02, is scheduled to launch in August, followed by GPS IIIA-03 in December.

PublicNTP is extremely excited to see GPS III finally get off the ground—pun absolutely intended. The number of instruments each A2100 bus carries will be a remarkable improvement over our current system designed between 1960 and 1980.

GPS III introduces three new civil signals as the IIIA and IIIF blocks are launched over the next couple decades:

• L2C – A frequency designed for clear commercial use
• L5 – A Safety of Life frequency designed to enhance the safety of aircrafts
• L1C – A secure frequency designed to facilitate interoperability between international satellites

Each of these advancements reinforce the technology infrastructure that the entire world has been utilizing for over 20 years. For PublicNTP, it results in additional resources to help ensure that a world with fully-synchronized time can be a reality.

The post GPS Generation Three Has Launched! appeared first on Public Time.

Terry lives out around D.C. but he and I met in Salt Lake after his family moved here from Maine in 1994. Kind of startling to realize he and I have known each other for a quarter of a century!

In June, Terry was in Salt Lake City, Utah visiting family. Whenever Terry’s in town, he and I make a point to meet up and get some quality face-to-face time — it’s amazing how much work can get knocked out in just a couple hours when you’re sitting at the same table.

Wylie Thomas, the founder of PXP, a development contracting shop located just north of Salt Lake that PublicNTP has worked with extensively, was kind enough to offer up the use of his office’s conference space for the meeting.

Terry, Wylie, Cody Deskins (who had to bail just before the picture was taken), Eric Evans, and I talked about the efforts we have going on the PublicNTP website, the Time Server App, and then did some brainstorming on some future projects that are in the pipeline.

Terry suggested that while he was in town, we take advantage of the mountains surrounding the Salt Lake Valley to put on some altitude, as he doesn’t get much of that in Virginia. One afternoon we headed up Little Cottonwood Canyon and took the tram to the top of Snowbird Ski Resort to check out the Salt Lake valley from the east, perched up at 11,000 feet/3,353 meters above sea level. A few days later we did a short hike from the top of Butterfield Canyon to examine the Salt Lake Valley from a west side vantage point at 7,915 feet/2,400 meters elevation. Check out more pics on Flickr from Snowbird and Butterfield Canyon.

One of the topics Terry and I talked about in depth was how PublicNTP can better engage with the broader time community.

Not unlike the physical distances between the PublicNTP board members, Terry and I agree that, as with any online community, the network time community has communication challenges. Creating opportunities to foster communication amongst all the people interested in network time is an important and ongoing challenge the group as a whole needs to constantly be working on.

If you are passionate about time as we are, please consider donating to PublicNTP!

The post F2F in Mountain Time! appeared first on Public Time.

PublicNTP has been working to improve our footprint in Africa since before the company was legally incorporated in the state of Virginia. The company has dubbed the search for improved company presence in Africa Project Ikenga, named after an African god of time.

The team has found Africa a challenging place to find reliable computing infrastructure at a price point that a nonprofit can sustain long term — and we’re not alone. Problems like having all our african servers hosted with a single company go offline the same night and finding concrete proof several months later that the provider went out of business with absolutely no warning is sadly common in a lot of developing parts of the world.

In early March, I was very glad to see Microsoft announce its Azure cloud datacenter in Johannesburg, South Africa had gone live. Of the hyperscale providers — Amazon, Microsoft, and Google (who altogether combined hold 75% of the total global cloud market share), Microsoft is the first hyperscale cloud provider to open a region in the entire African continent.

AWS has announced it will open a region in Cape Town, South Africa in early 2020. But by the time AWS goes live, Microsoft will have had its African cloud region live for a full year. And in our line of work, time is literally of the essence.

The Azure server in Johannesburg is still in early testing phases for the PublicNTP fleet, but early data is looking very promising as to its performance and accuracy. Expect more news soon!

New NTP server software: NTPsec

PublicNTP has run the NTP Reference Implementation software, maintained by Network Time Foundation, on all its servers to date.

While many UNIX users are used to seeing the ubiquitous and venerable ntpd in a process list, it is important to note that there are multiple mature software packages that will provide NTP service.

As part of the always-ongoing maturing of the PNTP server fleet operations, we decided as we launched the Johannesburg server, it was a good opportunity for us to start diversifying our NTP server software.

PublicNTP has closely followed the NTPsec project since its inception. NTPsec has a number of contributors, but is led by the well-known Eric S. Raymond (usually shortened to “ESR”).

After months of internal testing, PublicNTP was confident that NTPsec was entirely up to the challenge of running in the PublicNTP global fleet.

The “Community” in “Network Time Community”

I personally wanted to call out yet another example of the incredible good will in the network time community.

Back in 2017, PublicNTP had a response from the Network Time Foundation within 48 hours of contacting them to let them know PublicNTP had incorporated and was looking to use NTF software in PublicNTP deployments.

PublicNTP also had a very positive first encounter with the NTPsec project; within 24 hours of reaching out to make them aware PNTP was testing their software with the intent to deploy to production, we’d heard back from multiple members of the NTPsec team, all of whom were all very friendly, helpful, responsive, and encouraging.

We’re excited to be cultivating such connections and are eager to grow the community by providing easier NTP access for free to the public.

The post Touching Bases With The Team (African Expansion and More!) appeared first on Public Time.

Multi-Constellation

• Our first iteration of this architecture uses the Meinberg GPS180PEX, which can only see US Global Positioning System satellites.
• There are several competing Global Navigation Satellite Systems (GNSS), including the EU’s Galileo, Japan’s QZSS, Russia’s GLONASS, and India’s NAVIC.
• Using hardware that can process the signals from other constellations increases the robustness of the architecture, due to:
  • Greater availability of satellite-based time sources becoming available
  • Breaking the dependency upon a single time provider (in the case of GPS, the US Naval Observatory Master Clock which is the source of GPS time)

Multi-Frequency

• The first iteration of the architecture can only receive the GPS L1 Coarse/Acquisition (“L1 C/A”) signal.
• Making it possible for the architecture to receive timing information on at least two different frequencies (e.g., L1 and L5) allows the receiver to compute — and thus eliminate — the amount of ionospheric error, which is one of the largest contributors to GPS location/timing errors.

Indoor Reception

• Newer GNSS signals, such as GPS L5 and Galileo E5a/b, were carefully designed to be easier to detect as well as broadcast from the satellites at much higher power levels.
• The improved signal design and strength should make it easier to maintain GPS lock for L5 versus L1 C/A, especially in situations where L1 C/A typically struggles, such as when indoors.

Lowering Price

• At 2,700 USD, the Meinberg GPS180PEX GPS receiver is the single largest factor of the overall cost of the first iteration of the Public Time Open Architecture Stratum One platform.
• There are numerous articles on the internet about Raspberry Pi-based GPS receiver solutions that are significantly cheaper than the Meinberg hardware.
• One component typically lacking from the budget GPS receiver solutions is a precise oscillator to allow the system to “freewheel” with very low drift when the GNSS signal is not available.
• We’ve been looking into much cheaper solutions for both halves (GNSS signal reception as well as the precise oscillator). It’s likely that we would have to write an operating system driver to support our oscillator solution, but that’s a challenge the Public Time team is eager and well-suited to take on.
• The lower we can push the price point, the more people can take advantage of our work, which makes it a constant goal of the project.

There are many different avenues to improve our first iteration on this platform; these are just the first ones we noted and remembered to write down. 🙂

We are very interested to see where the network time community takes this, which will likely be well beyond what we have envisioned so far!

The post Public Time’s Open Architecture Stratum One – Part Five: Future Investigations appeared first on Public Time.