Hi!
I went down the rabbit hole of making something as simple as a ticker with live currency exchange rates (e.g. USD to EUR with minutes of time resolution), but it turned out to be a huge pain. Apparently there are zero providers with public APIs (unlike crypto listing sites which have several convenient API providers), so it comes down to web scraping.
Obviously the big sites (xe, Google finance etc.) protect against scrapers by being a nuisance, first and foremost by responding with 0.5-1MB of HTML garbage to each request. In addition, Google requires session-specific consent cookies for European IPs (replacing the old "consent=YES+" cookie), and xe.com returns outdated (non-live) currency rates when the gzip compression header isn't sent. Rainmeter's WebParser can't read compressed http responses and has no conditional logic to catch the consent prompt and create the required session cookies. But even if there was a way around, it's kind of preposterous to generate hundreds of MB of traffic per day just to display a few bytes of data in a ticker. The "proper" way (without paid API access) would be to run an actual headless browser engine to extract the temporary credentials from the js code for the API access used by the website frontend - but that's certainly far beyond the scope of the poor WebParser, though gzip support would be really really useful and simple enough to enable.
Finally I compromised and settled for Finacial Times data (https://markets.ft.com/data/currencies/tearsheet/summary?s=EURUSD), which is not really live but good enough, and wastes "only" ~100kB per request. Am I making it harder than it should be? Is there a suitable data source out there for "hobbyists" which I just haven't found?
I went down the rabbit hole of making something as simple as a ticker with live currency exchange rates (e.g. USD to EUR with minutes of time resolution), but it turned out to be a huge pain. Apparently there are zero providers with public APIs (unlike crypto listing sites which have several convenient API providers), so it comes down to web scraping.
Obviously the big sites (xe, Google finance etc.) protect against scrapers by being a nuisance, first and foremost by responding with 0.5-1MB of HTML garbage to each request. In addition, Google requires session-specific consent cookies for European IPs (replacing the old "consent=YES+" cookie), and xe.com returns outdated (non-live) currency rates when the gzip compression header isn't sent. Rainmeter's WebParser can't read compressed http responses and has no conditional logic to catch the consent prompt and create the required session cookies. But even if there was a way around, it's kind of preposterous to generate hundreds of MB of traffic per day just to display a few bytes of data in a ticker. The "proper" way (without paid API access) would be to run an actual headless browser engine to extract the temporary credentials from the js code for the API access used by the website frontend - but that's certainly far beyond the scope of the poor WebParser, though gzip support would be really really useful and simple enough to enable.
Finally I compromised and settled for Finacial Times data (https://markets.ft.com/data/currencies/tearsheet/summary?s=EURUSD), which is not really live but good enough, and wastes "only" ~100kB per request. Am I making it harder than it should be? Is there a suitable data source out there for "hobbyists" which I just haven't found?
Statistics: Posted by Novgorod — April 24th, 2024, 2:38 am — Replies 9 — Views 169