The Sorry State of Network Maintenance Notices

It’s nearly impossible to automatically parse network maintenance notices from internet providers. Let me tell you about a few ways providers mess this up!

I was driven to create this article after receiving a network maintenance notice, which my parsing code rejected. The problem today? The maintenance was to begin and end at 6:00 GMT. Even the most trivial maintenance activities take a minute or two, but this escaped their system and found its way to my parsing code. Sadly this isn’t unusual.

What will it take for providers to send maintenance notices that can be parsed by a mere mortal?

How can we solve this?

An expired IETF informational draft, Maintenance Notification Improvements Using iCalendar, attempted to solve this, but seems to have gotten bogged down in IETF processes. I hope to help get something like that actually standardized, so we can tell providers what we want from maintenance notices, rather than needing to accept the notices we get from them today. In the meantime, if you’re implementing maintenance notifications for a provider, I urge you to try to implement the draft above, ensure that your messages are able to be parsed automatically (think about things like selectors for finding maintenance in your HTML email notifications), and, in an ideal world, I could just call your provider’s API service and get a list of maintenance activities, with associated dates, status, expected impacts, and circuit IDs. This would make life easier for all of us.

I’ve also chosen to use an Open Source python library to parse maintenance notices, whenever possible. While this library is incomplete, it felt like a better service to the community for our common problem if my efforts helped to improve this library. I would love to see network providers write their own parsers for maintenance activities to be added to this library. You know how your messages are formatted, and won’t need to reverse-engineer them like the rest of us!

But, what’s the state today?

Here’s some examples of how things can go wrong:

• One provider uses a format that generally matches the IETF draft mentioned above. Except their iCal attachment is improperly formatted (they use a tab instead of newline to separate lines). Thus iCal parsing libraries can’t parse it.
• A few providers only send HTML emails, which tend to be harder to parse. They also seem to avoid giving IDs or names to HTML elements that contain useful information.
• One provider sends a two column table listing circuit IDs. Except when they want to send a three column table.
• Date formats can be entertaining. Pretty much every format imaginable. For instance, some providers send start dates like “01-JAN-2024 23-00 GMT.”
• Time zones! If you are lucky enough to get them, don’t expect them to match anything standard. You’ll see things like ET (US Eastern Time) and GST (Greenwich Standard? Time).
• Consistency! Just because a provider uses dates like 2024-01-02 one day doesn’t mean they will the next. It might be 1/2/2024 in their next maintenance notification. And good luck figuring out if that’s January 2nd or February 1st.
• One provider sends the circuit IDs in an attached CSV. Okay, easy enough. Except it’s not in UTF-8 format, but in UTF-16 format… And did I mention that it’s really a TSV, not a CSV (tabs, not commas)?
• A different provider uses base64 encoding of their plain text emails!
• Yet a different one adds a layer to this: They send a plain text part that is base64 encoded, with a MIME type of text/html, but other than the presence of <br> elements at the start of each line (yes, the start), aren’t HTML. Notably they don’t quote characters that need to be quoted in HTML. This causes a lot of HTML libraries to report that the HTML is invalid (because it isn’t really HTML).
• Speaking of base64, some providers use it…sometimes and sometimes not…for their messages.
• Speaking of HTML, one provider does, sort of, quote characters HTML considers special, even when they shouldn’t! So you have attributes like class=\"windowStart\" – and, yes, you are correct if you think those backslashes don’t belong there!
• How do you notify when a maintenance activity is complete? It turns out that you do that by using random phrases to mean completion.
• Speaking of random phrases, one provider talks about “declination.” For what it is worth, that’s the angle of something in space relative to a plane through the earth’s equator. I suspect they think it’s a fancy way to say “decline” or “reduce.”
• One provider has an API, which sounds great-it even exposes the notifications in the API. However, the API doesn’t expose these as structured data you can easily use to locate circuit IDs, times, and expected impacts. Rather, it just lets you download the same HTML they email you.
• Providers swapping start and end times is common (I.E. a two hour long maintenance might be scheduled to start at 6:00 PM and end at 4:00 PM).
• Character sets! If you aren’t using Unicode and a Unicode encoding, at least tell me what character set I should use to interpret your high codepoint characters!
• Whitespace! And it’s where you won’t expect it.

I don’t mean to complain too much. It can be worse: Some providers don’t even send a notice!

What can you do?

If you work somewhere that sends out maintenance notifications, make sure they can be parsed by machines. At the very least, make sure that you can easily use XPath to locate interesting information, if using HTML. Generate your notices via a machine that checks for common errors (dates that can’t be parsed, invalid circuit IDs, start times before end times, start times in the past, no circuits listed).

Add a parser and provider to the Circuit Maintenance Parser library so your customers don’t have to do that. You’ll also discover gaps in your notifications this way! Consider using the iCal attachment format documented in the expired internet draft (since this is the best we have right now, even if expired), but keep updated and update your notices once an actual specification is created!

You’ll make your customers happier. Trust me.