All quiet on the Corona front 2020-04-20
Hope you are all doing ok in these difficult times. Here at MET Norway it's business as usual. Most of us are working from home, but since the API team already did this for half the week there's not much difference. Here's a summary of the changes so far this year:
End of Life'd products
The following products have been deprecated and will be removed on
15 May 1 June 2020:
The following have also been deprecated, probable EOL times as indicated:
Version 2.0 is nearing completion, the only major feature missing is documentation. We have decided to split the output into three different endpoints:
- default (new, plain vanilla JSON)
- compact (a lite version of JSON with less bloat)
- classic (good old fashioned XML, to be used responsibly)
Also, the unit for temperature has been changed in JSON from "C" to "celsius" to conform with CF standards.
We are also happy to launch a new set of corresponding weather icons. Unlike the former numerical codes in 1.9, the 2.0 "symbol_code" variable is based on a mnemonic code and correspond exactly to the icon filename (minus extension). All icons are available in PNG, SVG and PDF formats, and can be downloaded as a gzipped TAR archive:
In addition you can also get a table matching the new and old symbol codes, as well as explanatory texts in English and several variations of Norwegian:
Radar have been upgraded to version 2.0, with new parameters, new areas, new sizes and a new projection. GIF animations will still be producted for the time being.
The new version 2.0 delivers both SIGMETS and AIRMETS, but now also includes wind shear warnings. You can now ask for each type separately, but by default you will get all in the same response. Also the long-standing bug where warnings would be included twice have finally been fixed.
This was previously part of Upperwindweather, but has now been launched as a separate product.
As mentioned last year we have deprecated the LocationforecastLTS product, which will be retired some time after we have launched the new Locationforecast 2.0. In the mean time, the background model for the Nordic area (MEPS) have been replaced with a new, improved model (CMEPS) which is run every hour.
Unfortunately, some little used parameters are no longer present in the new model, which means that some slight changes have been made to locationforecastlts:
- the parameter areaMaxWindSpeed was removed
- there will no longer be available time intervals backward in time
These changes are only be visible for the Nordic area (Norway, Sweden, Denmark and Finland). The rest of the world (which uses the EC model) remains unchanged.
Some of you may have noticed that about half of the municipalities in Norway got new codes ("kommunenummer") this year. This was implemented in MetAlerts this March. As you cannot search on municipalites yet (only counties) rhis does not affect the API interface, but if you are extracting kommunenr from the CAP files you should search for both old and new numbers.
In March we changed the file storage backend for non-dynamic products. This should result in faster response times and greatly reduce the occurence of delivering stale data.
Future plans for 2020
- Icemap/2.0 (possibly with GeoJSON)
- Probabilityforecast/2.0, Nowcast/2.0 and Oceanforecast/2.0 in JSON formats
Ho, ho, ho and some API stocking fillers 2019-12-20
We hope you all have been good developers this year, in which case you can look forward to some new improved products next month (for those who have been bad we will also be taking some away).
On December 3rd we launched the new backend for locationforecast/1.9. In connection with that some of our users have noted that the topography for Norway is not as accurate as before. This is because we use a terrain model with a slightly coarser resolution. This does not affect the quality of our temperature forecasts. However, due to requests from our users we will soon return to a higher resolution topography for Norway.
Next year we also plan to introduce a higher resolution topography for our global forecasts. This will make the forecast more accurate when not using the recommended "msl" parameter to indicate elevation, which is hard to implement e.g. in mobile apps.
We are also working hard on finalizing the JSON format in 2.0, as well as improving the weathericon symbols so that you won't have to call sunrise to know if to use a night or day version.
Due to popular demand, we have decided to continue GIF animation in radar/2.0 which will launch on Feb 3rd. Everything else will be different though, included sizes (only one), areas (previously called radarsites), types (more options) and formats (all static images are now in PNG).
More changes on Feb 3rd
Next year the sigcharts in upperwindweather will be moved to a new product, possibly with some new charts to follow. The upperwindweather product will then be deprecated, and eventually removed from the public api.
We will also be launching a new version 2.0 of icemap with more and better maps of the Arctic. The old version 1.5 will continue running until March.
We have also decided to retire the little-used lightning product on api.met.no. Since you now can download lightning data more or less in real time from Frost, there is little use for 5 min old data from the API. The data will continue to be delivered in UALF format as before. You can test out the new product here (remember to register for a developer key).
Finally the radar/1.5 and uvforecast products will be end-of-lifed on May 15th, to be replaced by the improved radar/2.0 and locationforecast/2.0 respectively.
Changes to Textforecast and Locationforecast 2019-11-13
We will shortly be making some changes to the existing versions of Textforecast and Locationforecast.
The seahigh and seabank products will be replaced with a new, auto-generated product (named sea) which will be available on December 3rd (possibly earlier). The old forecasts will continue production until February 1st, after which requests will be returned the new sea forecast instead.
We will shortly be changing the backend for the locationforecast to a new, cloud-based service. For the immediate future there will be very little change for the users except for higher performance. There are some minimal changes to the version 1.9 output, in short:
Only 1- and 6-hour forecasts will be supported, the 2- and 3-hour intervals will be removed.
We have added precipitation minvalues and maxvalues for the whole forecast for the Nordic and Arctic regions, not just the first two days:
The EPS, LOCAL and EC* models in the meta header are no longer relevant. We now how only one model (met_public_forecast), with a single
The global forecast (outside the Nordic and Arctic regions) will be shortened by 6 hours. As a compensation it will update every 6 hours, instead of every 12 hours as previously.
If you experience any problems due to these changes, we suggest temporarily switching over to LocationforecastLTS until you can make the necessary changes.
As Locationforecast and LocationforecastLTS currently runs on the same old
backend and is virtually similar, we have decided to end-of-life this product
as it is impossible to guarantee the same output on the new backend. This
means LocationforecastLTS will be officially deprecated once the new model
is in production, but will not be terminated until the end of February 2020
at the earliest (which should be plenty of time to change your
implementation). Since the new backend is much faster and has a more correct
forecast, we suggest changing over to I
Plans for next year
Some of the planned changes for the next 6 months include:
In the long term we will be launching a new 2.0 version, which features:
- a new, proper JSON format (instead of conversion from XML)
- probability values (as in the current probabilityforecast)
- UV index data (replacing the current UVforecast product)
- more understandable weathersymbol codes, hopefully without the need to contact sunrise
At the same time we will be deprecating the Probabilityforecast and UVforecast products since the data will now be available in Locationforecast instead.
UVforecast: These are produced by external parties, and we experienced distribution problems last summer. Since UV data will be available in locationforecast/2.0, the UVforecast product will be terminated after Easter 2020. UV maps should still be available from external distributors.
ExtremesWWC: This product will be replaced by a similar feature on our observations and climate API frost.met.no.
We do not have exact dates for the above changes, but will return with updates once we have more specific information.
I know what your API did last summer 2019-08-21
Making an API is a thankless task. Most users aren't aware they are using it, and you rarely hear from the developers unless something isn't working. While our weather site Yr is hugely popular, our API doesn't get much publicity. So when we finally get some exposure, it's sad to find only negative publicity, and misguided criticism at that. But before addressing the issues, let's watch the presentation (starting about 8:44 into the video).
"...a company called met.no, seems to be a weather type service..."
To start with, met.no isn't a company, it's the Norwegian Meteorological Institute – a state-owned, non-profit government organisation which has as one of its missions to give free (both as in beer and in freedom) weather data to the whole world. So far we've managed to do this without any developer registration or other conditions than nicely asking not to overload our services. As a consequence we have no clue about most of the thousands of apps and sites using our services, and no way of contacting them unless they subscribe to our mailing list or read the API front page.
From the interview one would get the impression we are making apps ourself using unsecure communications. Such is not the case – the only app we have some influence over is the Yr weather app, which doesn't speak with our API at all. So already the (admittedly valid) criticism is targeted at the wrong party.
"... it was making an HTTP request, totally normal"
Disregarding that "it" in this case is an app we have no control over, we have actually supported (and recommended) HTTPS for many years (although in the old API this wasn't default because of compatibility issues). Since the rewrite and launch of WeatherAPI v.3 (which came out in beta in 2016 and was gradually phased in on a per-user basis over the next year) we have only supported HTTPS. All unencrypted HTTP traffic is either being redirected or blocked (more on this later).
To this day we still get complaints about this, e.g. when no longer being able to use Squid as a local caching proxy, or using client libraries which don't support SSL/TLS. Incredible as this may seem in 2019 it is still an issue, and when your application is a watering plant run by a Saia-Burgess PLC controller there is no sofware upgrade path.
About half of the requests to api.met.no are either throttled, blocked or invalid. Of the latter, most are for products which were removed years ago. We're still getting requests for locationforecast/1.8 which was removed in 2014(!). Most developers don't bother checking the response status codes, so lots of sites and apps go on blindingly ignoring 304, 404 and 429 errors for several years. And since many users don't update their mobile apps until they have to, we're getting a lot of legacy traffic, included unencrypted HTTP which we're unable to prevent people from sending us.
As an example, let's look at the HTTP traffic from Android (pre <5.0) apps where the developer haven't read the TOS and uses the default "Dalvik" User-Agent header which amounts to 13% of the total unencrypted HTTP traffic. These are the amount of requests sorted by status code between 12 and 13 today (UTC):
As the table shows, the unencrypted traffic amounts to a whooping 43% of the total identifying only as Android (and an obsolete version at that). How much of this actually handles the redirect is difficult to see as we don't track users, but we find it hard to believe that 76% percent of the HTTPS traffic comes from redirects. This means that a large portion of clients never follow the redirect.
Of those who use HTTPS, an incredible 55% of the HTTPS requests are being throttled for too much traffic or missing identification, while only 43% get a meaningful answer. The remaining requests (apart from a handful of timeouts) are either for products which been discontinued (sunrise/1.1 is still very popular) or they can't manage to construct a valid URL. That's a lot of developers who can't be bothered to RTFM.
"... trying to figure out when the sun goes up"
Now, this is where it gets interesting. Most of the HTTP traffic is actually for locationforecast; sunrise doesn't count for more than about 5%. Of this, 5 out of 6 sunrise requests are actually zombie apps calling version 1.1, which was discountinued in February and returns a 404. (Incredibly we also get some for version 1.0 which was EOL in 2016!)
Still, that leaves about 1% of HTTP traffic for sunrise/2.0, which has never worked with unencrypted HTTP. Every piece of documentation, example code and tests we have published have been using HTTPS, so why are people still using HTTP? These requests are (if identified at all) coming from Android, so either there is something amiss with the Dalvik HTTP client stack, or developers are intentionally breaking security by rewriting our examples from "https:" to "http:". As the saying goes, it's hard to make stuff fool-proff because the fools are so smart.
"Latitude and longitude, that's harmless, right?"
Many of the requests do indeed contain geocoordinates. However, the vast majority of these originate from servers located in data centers totally remote from the geolocation in the request. Even when an app enable location services on your phone, we have no way of knowing if the coordinates are your actual GPS location or if you're just checking the weather for your next trip to Paris. We're still getting ~25,000 different IP addresses from all over the world downloading the weather forecast for a cornfield in Poznan, PL every 5 minutes, amounting to 3.15% of the total api.met.no traffic.
"... but you've got the MAC address [...] and lat/lon"
I confess this claim has us stumped, and after putting our collective heads together we still can't figure out what they're getting at. Sure, the MAC address is an identificator which can be used to link with personal data, but that is never transmitted to us (what the app developers are doing with your personal data you must ask them about).
Now, the only third party getting access to both your MAC address and your unencrypted traffic is either the owner of your Wifi access point/router or your phone operator. Admittedly, both these can be spoofed and your traffic intercepted by evil middlemen. However, in both cases they can already pretty much tell your geolocation even without snooping your data traffic. When a phone is connected to (in this case) a 800mW faked wifi access point and indoors it is pretty much given that the person is within a 50 m radius. In fact this is how shops track customer behaviour, logging how much time each person spends before each isle. Now, having the GPS coordinates in addition would probably give you a better resolution, but then GPS reception isn't great inside buildings anyway.
"So now we can track that person when their phone requests updates"
Sure, the app developers can (and probably will) track you, and the same goes for your phone company or free Wifi provider. But that has nothing to do with HTTP encryption, and very little to do with geolocation (which they know already).
Now, you might ask why we didn't go directly for HTTPS in the first place. At the time api.met.no was lauched in 2007, this was not usually an option; Facebook didn't offer HTTPS until 2011, and many client libraries did not support it. While this now has changed, our main priority has always been backwards compatibility, something we're proud of having had for 12 years now, even though individual products and versions have come and gone.
As mentioned earlier, all of the unencrypted HTTP traffic is either blocked or redirected, in the latter case to a similar HTTPS URL. Now, since we're mirroring the original URL in the Location response header, it can be argued that we still "support HTTP" since legacy apps can still go on working if they handle the 301 properly. If they instead got a 404, the theory goes, they would see the error in their ways and change their code to use HTTPS instead.
Unfortunately, in practice that is not the case. The sites topping the list of HTTP have all been blocked for abuse and get a 403 Forbidden response. They have been blacklisted for years, but still continue to hammer our servers with 14k req/hour each with no end in sight. For these large websites there is no connection between the IP address and the geolocation received by us, so any "leak of personal data" here would be between the browser and the website if they're not using HTTPS.
If we can conclude that policing obsolete websites is futile, then this goes double for obsolete mobile apps where it doesn't matter what the developer does unless the user updates the app. As can be seen from the previous table, a lot of the traffic isn't only against services which no longer exist, but are also more than two years old and run on obsolete versions of Android. There is not much hope that these apps will be updated any time soon.
Anyway, as a precaution we tried disabling the redirect to HTTPS to see what happened. Not only did we get a lot of support tickets from developers who couldn't figure out what was wrong, it also broke the website. Whereas you previously could type e.g. "api.met.no/apis.json" in the browser location bar and get JSON back, the browser actually defaults to unencrypted HTTP which no longer works. Instead you actually have to type "https://" before the hostname. Can you think of any other sites in 2019 where this is necessary? Me neither.
In fact, all major sites we have tested (e.g. Google, GitHub, Paypal, eBay) redirect HTTP requests to corresponding HTTPS URLs, which is also recommended in all best practice documentation we have found, including the HTTP Strict Transport Security. Here is an example of GitHub doing exactly the same thing:
$ GET -Se http://api.github.com/users/octocat/orgs GET http://api.github.com/users/octocat/orgs 301 Moved Permanently Connection: close Date: Fri, 23 Aug 2019 11:08:56 GMT Location: https://api.github.com/users/octocat/orgs Content-Length: 0 Client-Date: Fri, 23 Aug 2019 11:08:57 GMT Client-Peer: 18.104.22.168:80 Client-Response-Num: 1 GET https://api.github.com/users/octocat/orgs 200 OK
So we've rolled back the change, and until we can find a way to differentiate zombies from actual live humans we're keeping it that way.
The bottom line
So, the next time you hear someone boldly stating that
"APIs leak personal data"
you would be wise to interpret that as
"abandoned zombie apps, written by unknown parties for a defunct API, who send unsolicited, not-very-personal data to non-existent services, can be intercepted by hackers who will learn nothing they don't already know, and there is nothing the API owner can do to prevent it".
Admittedly that won't make any headlines at a security conference. Instead,
if there's anything to learn from this, it's that most traffic against APIs
is actually noise, which you'll just have to learn to deal with. But
admittedly that doesn't make for good soundbites on
More changes to Radar and Radarlightning 2019-07-02
Due to increased workload by the new 5 min radar interval, we have had to reduce the number of different images available. In short, generating GIF animations from existing static GIF images continues as before. However, due to the 50% increase in images and 33% shorter production window, generating animations from PNG is no longer possible and has been terminated. We apologize for the lack of advance notice, but we had not foreseen this would be a problem.
Since we know this will cause existing functionality to break, we've come up with a temporary solution where we substitute the missing animations with similar products. In most cases the pixel dimensions will be different, but this can be scaled in the browser. Also note that content=image and content=animation now will return totally different images, so be wary of using size=small for static thumbnails for the animations.
reflectivity_hires and rainrate_with_phase
All animations of type reflectivity_hires and rainrate_with_phase will be substituted with type reflectivity. Where radarsite indicates an area these will be identical, but single radars will be substituted to the closest area as per the following table:
All requests for lightning animations (including the Radarlightning product) now return the "norway" image. All other animations have been terminated.
We have some images currently used internally which could be substituted for each individual area, but they are currently not available on the API. We will be working on a possible replacement later this summer.
"medium" and "large" size static images
The little used "medium" and "large" sizes have been terminated. All requests for these sizes now return the "normal" image. If you scale this in the browser it should not be noticable to the end users. This does not affect radarlightning which already uses only "normal" size.
Also, the small GIF animations haven't been produced for a long time and will be removed from the API. This means only the following variations will be available in the future:
|image||normal (default), small|
Again, we're sorry for the inconvenience, but hopefully the faster radar scan rate should compensate for the reduced size options.
We strongly suggest you consider dropping GIF animation as soon as possible,
out GIF completely
by the end of 2019 sometime during early 2020.
Changes to Radar, Nowcast and other products 2019-06-07
From Wednesday June 12th, the radar images will be produced every 5 mins.
- new product "aviationforecast", see below
- new beta product "volcanicashforecast"
- sunrise now returns moon position and phase for each day
- routeforecast have several new routes
- upperwindweather now has a status function showing image update and transmission errors (if any)
Spotwind 1.0 has been replaced by 1.1 which delivers data for 24 hours instead of 6. Version 1.0 will expire 2019-09-02.
ExtremesWWC is expected to be removed some time in the near future, to be replaced with a similar feature in our Frost API for observations. We will come back with further news when this is available.
Textforecast 1.6 has been deprecated, and the land and sea forecasts have been re-implemented in 2.0 with a new, cleaner XML format which can now be validated. The aviation forecasts have been moved to a new product "aviationforecast" but remains otherwise unchanged. Since the migration is now complete we will terminate version 1.6 on 2019-09-02.
New API products 2019-03-04
We will shortly launching some new products, mainly of interest for Norwegian users. As described earlier, some obsolete products will also be removed.
We have started the process of cleaning up the old text forecasts, with a new XML format (which can be validated), more consistent data structure and clearly defined areas. All of new forecasts share the same format and schema, including:
- coast_en, coast_no
- seabank_en, seabank_no, seabank_wmo
- seahigh_en, seahigh_no, seahigh_wmo
To aid in transition from textlocation, all areas are now identified with an id, which can be used to look up the corresponding geographical coordinate polygon. These files can be downloaded from the /areas endpoint, but we hope eventually to be able to include them directly in the forecast.
We have removed the "land" product (use "landoverview in 2.0) as well as "seaoslofjord" (use "coast_no" and "seabank_no" in 2.0).
For some time we have been working on air quality forecasts in cooperation with several other public services. This is expected to go out of beta soon.
This is a new image service for aviation, primarily offshore helicopter flights.
This product will expire March 15th. Please use the "landoverview" product in textforecast along with the area files.
This service hasn't delivered any data for months and will expire on March 15th.
This product will expire
on April 1st, when we are turning off the WSklima service
(delayed until further notice).
There might be a similar service on frost.met.no in the future.
Discontinuation of text forecasts 2018-12-21
As mentioned earlier we are discontinuing some little used products from api.met.no, here is a complete list:
On February 1st we will remove the following products from the textforecast service:
the route_fbno products (these will still be available for professional users via NAIS which everybody seem to use anyway)
the landday/long, seabankday1 and metkyst products (which are used to generate more comprehensive forecasts)
the iga_fbno43, easter and miskred products (which haven't been produced for years)
We have checked the logs and all of these have little or no traffic, so we don't expect them to be missed. This is the full list of discontinued forecasts:
easter, landday0, landday1, landday2, iga_fbno43, landlong, metkyst0, metkyst1, miskred_north, miskred_south, route_fbno69, route_fbno70, route_fbno72, route_fbno74, route_fbno75, route_fbno76, route_fbno77, route_fbno78, route_fbno79, route_fbno80, seabankday1.
Also, on March 1st the textlocation service will be discontinued. We will also remove the "land" product from the textforecast service. There will be another forecast product to replace those two products, details of which are to follow later.
At the same time, the product "seaoslofjord" containing forecasts for Skagerrak and for the Norwegian coast from the Swedish border to Hordaland will be discontinued. Forecasts for Skagerrak can be found in the "seabank" product, while forecasts for the complete Norwegian coastline can be found in the "coast" product.
We will soon also remove the errornotifications service, which is no longer producing any data. This will be replaced by specific status messages for each product.
Due to licensing issues, problems with authentication and abuse of the service we have been forced to suspend delivery of non-restricted geosatellite images to the general public. Currently there are only 8 unrestricted pictures from a total of 2162 which we are allowed to distribute freely, but we have seen repeated attempts to download images which are clearly documented as not publicly available. Until further notice the service will only be available for our own sites yr.no and halo.met.no.
On Monday May 14th, 2018 from 07:00 to 13:00 UTC we will be running at 50% capacity due to replacement of the cooling system in one of our data centers. All services are expected to continue running, but latency times and timeout errors may increase due to reduced capacity.
Also, as has been noted earlier, the domains api.yr.no and beta.api.met.no will be turned off on June 11th, 2018. Please switch your systems to use api.met.no instead.
Migration completed 2018-02-26
As most of you probably have noticed, we upgraded api.met.no about A week ago to the new version. After some initial performance problems we managed to tune the caches so it is now handling traffic well.
Some users have reported problems accessing the API, mostly related to
not handling HTTPS and/or redirects. If you are getting "Failed to get
data from weather api. Reason: end tag name
</body> must match start tag
<hr> from line 5" this means you are trying to parse the HTML
message as XML instead of following the 301 redirect. We are also
working on recommendations for users using HTTPS via local proxies.
Finally, we have noticed there is a lot of production traffic against beta.api.met.no (which was only to be used for testing) as well as api.yr.no (which has been deprecated for years). We would like to inform you that these hostnames will be removed within a few months, so please update your applications accordingly.
Migration of api.met.no 2017-12-12
- Use HTTPS only! Unencrypted requests are no longer supported
- Honor redirects (301 and 303), which is used e.g. from HTTP to HTTPS
- Semicolon as parameter separator is now deprecated, use ampersand instead
- Make sure you have a unique, identifiable User-Agent string
Otherwise the main difference is a lot of new features and bugs that have been fixed. For more information on the changes to the interface, please see the release notes for version 3.
The migration is scheduled to take place some time during the first half of January 2018.