Code

Use Lazy Loaders#1536

Code

Merged

elronbandel merged 21 commits intomainIBM/unitxt:mainfrom lazy_loadHFIBM/unitxt:lazy_loadHFCopy head branch name to clipboardFeb 10, 2025

+272-283

Commits

Commits on Jan 27, 2025

• try lazy loadHF first

  Show description for 242d1b9
  dafnapension

  committed
  242d1b9

  Copy full SHA for 242d1b9

• reduce benchmark profiling to generating the dataset only. Not inferring (that is dome mocking anyhow) and not evaluating (of the mocked results). add trust_remote also to load_dataset_builder

  Show description for bc772b8
  dafnapension

  committed
  bc772b8

  Copy full SHA for bc772b8

• try procrastination for load csv too

  Show description for b9c3c4f
  dafnapension

  committed
  b9c3c4f

  Copy full SHA for b9c3c4f

• added split cache for the generators, and log limit once per data and increase loader cache

  Show description for 39a73a4
  dafnapension

  committed
  39a73a4

  Copy full SHA for 39a73a4

• make sklearn loader too - a lazy loader

  Show description for 709ac74
  dafnapension

  committed
  709ac74

  Copy full SHA for 709ac74

• adjust to new readers for csv

  Show description for a4121c7
  dafnapension

  committed
  a4121c7

  Copy full SHA for a4121c7

Commits on Jan 30, 2025

• Merge branch 'main' into lazy_loadHF

  elronbandel

  authored
  e741ac2

  Copy full SHA for e741ac2

Commits on Feb 3, 2025

• Merge remote-tracking branch 'origin/main' into lazy_loadHF

  Show description for 8ce505d
  elronbandel

  committed
  8ce505d

  Copy full SHA for 8ce505d

Commits on Feb 4, 2025

• Enhance LoadHF class to support optional splits and improve dataset loading logic

  Show description for 24d8f49
  elronbandel

  committed
  24d8f49

  Copy full SHA for 24d8f49

• Refactor LoadHF class to improve dataset loading and implement limit on yielded instances

  Show description for 151218d
  elronbandel

  committed
  151218d

  Copy full SHA for 151218d

• Refactor LoadHF class to streamline dataset loading and enhance split handling

  Show description for 42e1ad3
  elronbandel

  committed
  42e1ad3

  Copy full SHA for 42e1ad3

• Merge branch 'main' into lazy_loadHF

  elronbandel

  authored
  a83be91

  Copy full SHA for a83be91

Commits on Feb 6, 2025

• Merge branch 'main' into lazy_loadHF

  elronbandel

  authored
  493b28c

  Copy full SHA for 493b28c

• Remove unused import and update line number in secrets baseline

  Show description for 27f9c4f
  elronbandel

  committed
  27f9c4f

  Copy full SHA for 27f9c4f

• Refactor load_data method to simplify error handling and remove unnecessary cache checks

  Show description for 294eabf
  elronbandel

  committed
  294eabf

  Copy full SHA for 294eabf

Commits on Feb 10, 2025

• Merge origin/main

  Show description for 9ba5f1b
  elronbandel

  committed
  9ba5f1b

  Copy full SHA for 9ba5f1b

• Refactor loaders to implement LazyLoader class and update load_iterables method for improved streaming support

  Show description for 931138f
  elronbandel

  committed
  931138f

  Copy full SHA for 931138f

• Merge remote-tracking branch 'origin/main' into lazy_loadHF

  Show description for d21f16e
  elronbandel

  committed
  d21f16e

  Copy full SHA for d21f16e

• Update exception handling in test_failed_load_csv to catch general exceptions

  Show description for 85c2cab
  elronbandel

  committed
  85c2cab

  Copy full SHA for 85c2cab

• Refactor LoadHF class to streamline data loading and enhance error handling

  Show description for 692297c
  elronbandel

  committed
  692297c

  Copy full SHA for 692297c

• Merge branch 'main' into lazy_loadHF

  elronbandel

  authored
  102034b

  Copy full SHA for 102034b