Yahoo stopped to allow .historical() request : unauthorized error #795

stouch · 2024-09-07T12:02:41Z

Bug Report

Describe the bug

.historical() requests now fails with "User not logged in" error. https://www.reddit.com/r/sheets/comments/1farvxr/broken_yahoo_finance_url/

The text was updated successfully, but these errors were encountered:

gadicc · 2024-09-07T12:08:49Z

Thanks, we were tracking this at #349 (comment) but going to move here to make this easier to find (as a recent issue). My original response:

Mmm, on the Yahoo page for this, e.g. https://finance.yahoo.com/quote/TSLA/history/, I'm pretty sure there used to be a "download" link that would let us retrieve this data as CSV. I don't see that link anymore. Maybe it still appears if you're logged in? I didn't try as I don't have a Yahoo account.

In any event, all the data on the page itself (vs from a "download" link), seems to come from the chart API now. So, I guess our options are:

See if this is temporary and will fix itself again
See if it's possible if you're logged in, adapt the library to support Yahoo credentials
Probably best option, move over to the chart API (although, it would be nice if we adapted historical() to do this for the user with no code changes - I think we should do that but I can't commit to a time yet, maybe someone else will have a chance to do a PR before I get it).

https://github.com/gadicc/node-yahoo-finance2/blob/devel/docs/modules/chart.md

technologyatyottoldotcom · 2024-09-07T16:54:46Z

{"finance":{"result":null,"error":{"code":"forbidden","description":"User is not subscribed to Premium or has invalid cookies"}}}

freudl · 2024-09-07T19:09:58Z

@gadicc

I'm pretty sure there used to be a "download" link that would let us retrieve this data as CSV. I don't see that link anymore. Maybe it still appears if you're logged in? I didn't try as I don't have a Yahoo account.

Seems to be gone for authenticated users as well.

dtslvr · 2024-09-07T19:48:05Z

Probably best option, move over to the chart API

I guess I was able to fix the problem in the short term at Ghostfolio by switching from historical() to chart() and transforming the result like this: https://github.com/ghostfolio/ghostfolio/pull/3737/files#diff-7be88e7782ecbd969f1651da9ab3b2fc9ec36a2963784030dc2cbcec79cae690R313

Verdant31 · 2024-09-09T19:01:04Z

Probably best option, move over to the chart API

I guess I was able to fix the problem in the short term at Ghostfolio by switching from historical() to chart() and transforming the result like this: https://github.com/ghostfolio/ghostfolio/pull/3737/files#diff-7be88e7782ecbd969f1651da9ab3b2fc9ec36a2963784030dc2cbcec79cae690R313

Worked for me, its a good temporary solution for those that are looking for this

stouch · 2024-09-10T08:15:18Z

Thanks ! According to https://github.com/ghostfolio/ghostfolio/pull/3737/files#diff-7be88e7782ecbd969f1651da9ab3b2fc9ec36a2963784030dc2cbcec79cae690R313, these are simple steps to temporary fix this issue :

Replace your .historical() calls by .chart() calls
Just use this function to convert the chart call result to historical result:

import { ChartResultArray } from 'yahoo-finance2/dist/esm/src/modules/chart';
import { HistoricalHistoryResult } from 'yahoo-finance2/dist/esm/src/modules/historical';

/*
// Original answer:
const convertToHistoricalResult = (
    result: ChartResultArray
  ): HistoricalHistoryResult => {
    return result.quotes;
}
*/

// ⚠️ As chart and historical types have some different behaviour, you may have to convert a little bit differently with : 

const convertToHistoricalResult = (
  result: ChartResultArray
): HistoricalHistoryResult => {
  return result.quotes
    .map((quote) => ({
      ...quote,
      open: quote.open || null,
      high: quote.high || null,
      low: quote.low || null,
      close: quote.close || null,
      volume: quote.volume || null,
    }))
    .filter(
      (dailyQuote) => dailyQuote.low !== null || dailyQuote.high !== null
    );
};

For type resolution, try to use "moduleResolution": "node", in your tsconfig.json to resolve yahoo-finance2/dist/... types.

In case you used .historical to get dividends, use this function instead of convertToHistoricalResult :

import { HistoricalDividendsResult, HistoricalHistoryResult } from 'yahoo-finance2/dist/esm/src/modules/historical';

// ...

const convertToDividendResult = (
    result: ChartResultArray
  ): HistoricalDividendsResult => {
    return result.events.dividends.map(({ amount: dividends, date }) => {
      return { date, dividends };
    });
}

gadicc · 2024-09-10T08:39:43Z

Hey all, just a big thanks from me to everyone above, for sharing all your findings, and most especially, your solutions and their verification, which is a great help to the community at a time when I'm tied down with other commitments.

I'll just note that switching to yf2's chart() API really is the correct path forward here. All I intend to do when I have a chance is implement the same transformers like you have above into historical() to ease things for other busy people, but will still include a deprecation notice and warning to move over to chart().

Thanks again everyone 🙏

mattpetters · 2024-09-10T15:37:14Z

Thanks ! According to https://github.com/ghostfolio/ghostfolio/pull/3737/files#diff-7be88e7782ecbd969f1651da9ab3b2fc9ec36a2963784030dc2cbcec79cae690R313, these are simple steps to temporary fix this issue :

Replace your .historical() calls by .chart() calls
Just use this function to convert the chart call result to historical result:

import { ChartResultArray } from 'yahoo-finance2/dist/esm/src/modules/chart';
import { HistoricalHistoryResult } from 'yahoo-finance2/dist/esm/src/modules/historical';

/*
// Original answer:
const convertToHistoricalResult = (
    result: ChartResultArray
  ): HistoricalHistoryResult => {
    return result.quotes;
}
*/

// ⚠️ As chart and historical types have some different behaviour, you may have to convert a little bit differently with : 

const convertToHistoricalResult = (
  result: ChartResultArray
): HistoricalHistoryResult => {
  return result.quotes
    .map((quote) => ({
      ...quote,
      open: quote.open || null,
      high: quote.high || null,
      low: quote.low || null,
      close: quote.close || null,
      volume: quote.volume || null,
    }))
    .filter(
      (dailyQuote) => dailyQuote.low !== null || dailyQuote.high !== null
    );
};

For type resolution, try to use "moduleResolution": "node", in your tsconfig.json to resolve yahoo-finance2/dist/... types.

In case you used .historical to get dividends, use this function instead of convertToHistoricalResult :

import { HistoricalDividendsResult, HistoricalHistoryResult } from 'yahoo-finance2/dist/esm/src/modules/historical';

// ...

const convertToDividendResult = (
    result: ChartResultArray
  ): HistoricalDividendsResult => {
    return result.events.dividends.map(({ amount: dividends, date }) => {
      return { date, dividends };
    });
}

Thanks for laying this out so plainly, refactoring a project now and this is a godsend

DreamSkyMXDan · 2024-09-10T19:16:24Z

Thanks for the temp solution. Does anyone know if historical() will be fixed? When will it probably be fixed? thanks

GiorgioBaz · 2024-09-15T05:48:43Z

Thanks ! According to https://github.com/ghostfolio/ghostfolio/pull/3737/files#diff-7be88e7782ecbd969f1651da9ab3b2fc9ec36a2963784030dc2cbcec79cae690R313, these are simple steps to temporary fix this issue :

Replace your .historical() calls by .chart() calls
Just use this function to convert the chart call result to historical result:

import { ChartResultArray } from 'yahoo-finance2/dist/esm/src/modules/chart';
import { HistoricalHistoryResult } from 'yahoo-finance2/dist/esm/src/modules/historical';

/*
// Original answer:
const convertToHistoricalResult = (
    result: ChartResultArray
  ): HistoricalHistoryResult => {
    return result.quotes;
}
*/

// ⚠️ As chart and historical types have some different behaviour, you may have to convert a little bit differently with : 

const convertToHistoricalResult = (
  result: ChartResultArray
): HistoricalHistoryResult => {
  return result.quotes
    .map((quote) => ({
      ...quote,
      open: quote.open || null,
      high: quote.high || null,
      low: quote.low || null,
      close: quote.close || null,
      volume: quote.volume || null,
    }))
    .filter(
      (dailyQuote) => dailyQuote.low !== null || dailyQuote.high !== null
    );
};

For type resolution, try to use "moduleResolution": "node", in your tsconfig.json to resolve yahoo-finance2/dist/... types.

In case you used .historical to get dividends, use this function instead of convertToHistoricalResult :

import { HistoricalDividendsResult, HistoricalHistoryResult } from 'yahoo-finance2/dist/esm/src/modules/historical';

// ...

const convertToDividendResult = (
    result: ChartResultArray
  ): HistoricalDividendsResult => {
    return result.events.dividends.map(({ amount: dividends, date }) => {
      return { date, dividends };
    });
}

This is amazing thank you so much <3

Emre4x0 · 2024-09-15T10:57:58Z

hello, does anyone know when the error will be fixed on .historical()? i will update my code to .chart() accordingly. if .historical() is to be fixed soon, i don't want to change my code. 🫠

sachin8606 · 2024-09-16T10:23:41Z

Historical is working fine in python script, for node js you can use chart() function

Previously it used the "download" API which is no longer available. See the above issue for more info.

gadicc · 2024-09-16T14:16:57Z

Thanks, everyone. Finally had a chance to work on this. As discussed, it's simply a convenience to map the requests to chart() for you... you'd be better using chart() directly. The first time you call historical(), you'll get a deprecation notice like this:

[Deprecated] historical() relies on an API that Yahoo have removed.  We'll map this request to chart()
for convenience, but, please consider using chart() directly instead; for more info see
https://github.com/gadicc/node-yahoo-finance2/issues/795.  This will only be shown once, but you
can suppress this message in future with `yahooFinance.supressNotices(['ripHistorical'])`.

There'll be an automated message here once the build is published. As this build contains a number of other big internal changes, early testing and feedback (even without using historical()) would be greatly appreciated. All tests are passing so we assume everything should work like usual, but, since we completely changed the way validation works, something might have slipped through. (For any such issues other than historical(), please open a new issue).

# [2.12.0](v2.11.3...v2.12.0) (2024-09-16) ### Bug Fixes * **chart:** fix and improve non-Date handling code for periods ([f1106c2](f1106c2)) * **circleci,readme:** use node:18.0.0 for tests, make clear in readme ([8365a90](8365a90)) * **historical:** map requests to chart() with deprec notice ([#795](#795)) ([c212df9](c212df9)) * **imports:** add import assertions required in node since 17.5.0. ([7d0deea](7d0deea)) * **imports:** specify ".js" on local imports for full ESM compat ([46ee6c1](46ee6c1)) * **index:** remove erroneously removed moduleExec import ([fe1a493](fe1a493)) * **notices:** historical() note, type improvement, onceOnly text ([c131e5c](c131e5c)) * **pkg/scripts:** still need `prepublishOnly` to run `build` ([4da1273](4da1273)) * **pkg:** add babel importAttributes plugin ([e41f65d](e41f65d)) * **pkg:** drop ts-jest, use [@swc-jest](https://github.com/swc-jest), { keepImportAssertions: false } ([a0e2847](a0e2847)) * **pkg:** use swc, remove babel (faster, fixes import assertions) ([110d28d](110d28d)) * **tests-modules:** use swc here too ([c7e849c](c7e849c)) ### Features * **notices:** notices lib, refactor, use for "yahooSurvey" ([#783](#783)) ([b4c016b](b4c016b))

gadicc · 2024-09-16T14:33:49Z

🎉 This issue has been resolved in version 2.12.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

gadicc · 2024-09-16T14:34:55Z

Resolved but leaving this issue open for feedback and discoverability.

gadicc · 2024-09-16T15:04:46Z

Oops, now I'm seeing some of those null errors that @stouch mentioned 😅 That fix will come next 😅😅

e.g. yahoo-finance historical 0P0000XTS7 '{"period1":"2024-09-12"}'

gadicc · 2024-09-16T16:31:44Z

Above issue fixed in 2.12.1. I re-used the old row null logic from before (and as described in docs/historical.md#null-rows), which skips when entries when everything is null, but throws otherwise and asks the user to report. In the last few years that's never happened but who knows, could come in handy in the future 😅

P.S. @stouch, in your fix you have { volume: quote.volume || null }. Although it's rare, volume can be 0, in which case this will make it null, which might be unintended, depending on your use case. (Quick fix, use ?? instead of ||. I've never seen 0 for the other fields but better safe than sorry 😛).

mtorregrosa · 2024-09-17T09:04:41Z

I've just updated to version 2.12.1, but I cannot compile because the following types are no longer exported:

import { HistoricalDividendsResult, HistoricalHistoryResult, HistoricalRowDividend, HistoricalRowHistory } from 'yahoo-finance2/dist/esm/src/modules/historical';
import { QuoteResponseArray } from 'yahoo-finance2/dist/esm/src/modules/quote';
import { CalendarEvents, DefaultKeyStatistics, Price, QuoteSummaryResult, SummaryDetail, SummaryProfile } from 'yahoo-finance2/dist/esm/src/modules/quoteSummary-iface';

Any ideas on how to solve it.

Thanks

@eddie-atkinson

Pre-typebox, we exported all interfaces & types, actually because this was required for schema generation. Now that we don't need that anymore, whether or not we should do this in the longterm is a conversation that deserves merit, however, since some users make sure of these exports, we need to export them all to avoid it being a breaking change. In `quoteSummmary-iface.ts`, renamed the schemas to have a *Schema suffix and exported types without that suffix, like in other modules. Similar small change like that in `quoteSummary.ts` too. cc: @eddie-atkinson

## [2.12.2](v2.12.1...v2.12.2) (2024-09-17) ### Bug Fixes * **moduleExec:** actually return coerced values :) ([94430d8](94430d8)) * **modules:** re-export types, conform to *Schema convention ([#795](#795)) ([4269556](4269556))

gadicc · 2024-09-17T11:55:21Z

@mtorregrosa, right you are, this was a mistake on our side when we changed over to the new type system. Hopefully fixed in 2.12.2, can you confirm?

And everyone, one other critical fix in this release is the broken coercion (e.g. you wouldn't get Date instances and correct "transformed" types where expected). This is fixed now too, sorry about that! 🙇

mtorregrosa · 2024-09-17T14:38:21Z

For me, seems all fine in version 2.12.2, except that the following types have dissapeared:

import { HistoricalRowDividend, HistoricalRowHistory } from 'yahoo-finance2/dist/esm/src/modules/historical'

mtorregrosa · 2024-09-17T17:12:27Z

Furthermore, types seem to have changed from previous versions.
A lot of numeric fields that previously where of type number, now are of type number | { raw: number }. For example: regularMarketPrice, beta, dividendYield, bookValue, etc.

gadicc · 2024-09-18T09:33:01Z

Thanks, @mtorregrosa, this is really helpful. Our apologies again for the hassle and our thanks for taking the time to report and for your patience. I'm moving the type issues to #797 so we can track them all in one place 🙏

gadicc · 2024-09-19T13:48:18Z

Another small issue with the conversion that I found, for adjClose, because:

In historical() it's adjClose
In chart() it's adjclose

We don't currently handle this correctly, I need to fix it.

## [2.12.4](v2.12.3...v2.12.4) (2024-09-20) ### Bug Fixes * **historical:** correctly map adjclose -> adjClose ([#795](#795)) ([d8851ec](d8851ec)) * **historical:** return coerced values ([#797](#797)) ([386bf82](386bf82))

gadicc · 2024-09-20T12:00:54Z

Above fixed / released in 2.12.14 🎉

stouch added the bug Something isn't working label Sep 7, 2024

gadicc mentioned this issue Sep 7, 2024

Unauthorized error when using .historical() #349

Closed

SurferSamuel mentioned this issue Sep 13, 2024

Redesigned UI Theme SurferSamuel/Stash#6

Merged

gadicc added a commit that referenced this issue Sep 16, 2024

fix(historical): map requests to chart() with deprec notice (#795)

c212df9

Previously it used the "download" API which is no longer available. See the above issue for more info.

gadicc added the released label Sep 16, 2024

gadicc mentioned this issue Sep 18, 2024

Type issues since >= 2.12.0 #797

Open

gadicc added a commit that referenced this issue Sep 20, 2024

fix(historical): correctly map adjclose -> adjClose (#795)

d8851ec

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Yahoo stopped to allow .historical() request : unauthorized error #795

Yahoo stopped to allow .historical() request : unauthorized error #795

stouch commented Sep 7, 2024

gadicc commented Sep 7, 2024 •

edited

Loading

technologyatyottoldotcom commented Sep 7, 2024

freudl commented Sep 7, 2024

dtslvr commented Sep 7, 2024

Verdant31 commented Sep 9, 2024

stouch commented Sep 10, 2024 •

edited

Loading

gadicc commented Sep 10, 2024

mattpetters commented Sep 10, 2024

DreamSkyMXDan commented Sep 10, 2024

GiorgioBaz commented Sep 15, 2024 •

edited

Loading

Emre4x0 commented Sep 15, 2024 •

edited

Loading

sachin8606 commented Sep 16, 2024

gadicc commented Sep 16, 2024

gadicc commented Sep 16, 2024

gadicc commented Sep 16, 2024

gadicc commented Sep 16, 2024 •

edited

Loading

gadicc commented Sep 16, 2024

mtorregrosa commented Sep 17, 2024

gadicc commented Sep 17, 2024 •

edited

Loading

mtorregrosa commented Sep 17, 2024

mtorregrosa commented Sep 17, 2024

gadicc commented Sep 18, 2024

gadicc commented Sep 19, 2024

gadicc commented Sep 20, 2024

Yahoo stopped to allow .historical() request : unauthorized error #795

Yahoo stopped to allow .historical() request : unauthorized error #795

Comments

stouch commented Sep 7, 2024

Bug Report

Describe the bug

gadicc commented Sep 7, 2024 • edited Loading

technologyatyottoldotcom commented Sep 7, 2024

freudl commented Sep 7, 2024

dtslvr commented Sep 7, 2024

Verdant31 commented Sep 9, 2024

stouch commented Sep 10, 2024 • edited Loading

gadicc commented Sep 10, 2024

mattpetters commented Sep 10, 2024

DreamSkyMXDan commented Sep 10, 2024

GiorgioBaz commented Sep 15, 2024 • edited Loading

Emre4x0 commented Sep 15, 2024 • edited Loading

sachin8606 commented Sep 16, 2024

gadicc commented Sep 16, 2024

gadicc commented Sep 16, 2024

gadicc commented Sep 16, 2024

gadicc commented Sep 16, 2024 • edited Loading

gadicc commented Sep 16, 2024

mtorregrosa commented Sep 17, 2024

gadicc commented Sep 17, 2024 • edited Loading

mtorregrosa commented Sep 17, 2024

mtorregrosa commented Sep 17, 2024

gadicc commented Sep 18, 2024

gadicc commented Sep 19, 2024

gadicc commented Sep 20, 2024

gadicc commented Sep 7, 2024 •

edited

Loading

stouch commented Sep 10, 2024 •

edited

Loading

GiorgioBaz commented Sep 15, 2024 •

edited

Loading

Emre4x0 commented Sep 15, 2024 •

edited

Loading

gadicc commented Sep 16, 2024 •

edited

Loading

gadicc commented Sep 17, 2024 •

edited

Loading