Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Route API using old cached data? #45

Open
addd45 opened this issue Oct 25, 2024 · 2 comments
Open

Route API using old cached data? #45

addd45 opened this issue Oct 25, 2024 · 2 comments

Comments

@addd45
Copy link

addd45 commented Oct 25, 2024

Sorry in advance if I missed this in the docs... I check the API doc and swagger page, but I was only able to get some clue via looking at the API code.

It seems the route API fetches the csv data and imports it into a redis db, which is what the route api then uses.
However, the caching period seems very slow. I frequently am seeing old / outdated routes, despite the csv source being updated.

I understand this is a free service so I don't want to give the impression that I am complaining :). However, I am wondering if this intentional behavior or perhaps a bug? If intentional, would it be unreasonable to request a daily refresh? Does the refresh but a heavy strain on resources? Do feeders get more up-to-date data?

More details below--

For example, as of today the route API for callsign "JIA5248" returns the previous route of SYR -> PHL. It was updated 5 days ago (10/20/24) to STL -> PHL.

route api response:

[
  {
    "_airport_codes_iata": "SYR-PHL",
    "_airports": [
      {
        "alt_feet": 421,
        "alt_meters": 128.32,
        "countryiso2": "US",
        "iata": "SYR",
        "icao": "KSYR",
        "lat": 43.111198,
        "location": "Syracuse",
        "lon": -76.1063,
        "name": "Syracuse Hancock International Airport"
      },
      {
        "alt_feet": 36,
        "alt_meters": 10.97,
        "countryiso2": "US",
        "iata": "PHL",
        "icao": "KPHL",
        "lat": 39.871899,
        "location": "Philadelphia",
        "lon": -75.241096,
        "name": "Philadelphia International Airport"
      }
    ],
    "airline_code": "JIA",
    "airport_codes": "KSYR-KPHL",
    "callsign": "JIA5248",
    "number": "5248",
    "plausible": 0
  }
]

you can see that the route was updated 5 days ago @ https://github.com/vradarserver/standing-data/blob/main/routes/schema-01/J/JIA-all.csv

@addd45 addd45 changed the title Routes cached? Route API using old cached data? Oct 25, 2024
@iakat
Copy link
Member

iakat commented Feb 3, 2025

the api gets the csv from https://vrs-standing-data.adsb.lol/ so if that update breaks it will get outdated data. perhaps that is what you saw.

@addd45
Copy link
Author

addd45 commented Feb 3, 2025

The API does not pull from that csv real time - which indeed makes sense because that would be incredibly resource intensive to parse a csv upon each request. Thus it is cached into a redis(?) cache on certain intervals so that the APIs pull data from this cache.

My conern here is the frequency of the cache updates. If I understand correctly, the csv source is updated at least daily as new routes are constantly updated. Realistically, for this API, once daily would seem like a good balance or starting point, but any longer than that will surely yield invalid/expired data.

In my original post the API was returning results from a csv file over 5 days old, which means the API is not properly reloading the cache timely enough for the overall goal that the API is designed for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants