Skip to content

Commit

Permalink
fix: changes max_queue to 0
Browse files Browse the repository at this point in the history
The term fetchers were not retrieving all of the terms from solr because
the thread queue was overloaded. The number of threads is bounded for
these processes and it's important that all of the terms are collected,
so this value is set to 0 which means infinity.

Also changes ThreadPoolExecutor to FixedThreadPool in
authority_browse term_fetcher so that they match.
  • Loading branch information
niquerio committed Jul 26, 2024
1 parent c8e75de commit c88b9c3
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 8 deletions.
14 changes: 7 additions & 7 deletions lib/authority_browse/term_fetcher.rb
Original file line number Diff line number Diff line change
Expand Up @@ -84,14 +84,14 @@ def load_batch(batch)
end
end

# @retrun Concurrent::ThreadPoolExecutor
# @retrun Concurrent::FixedThreadPool
def pool
@pool ||= Concurrent::ThreadPoolExecutor.new(
min_threads: @threads,
max_threads: @threads,
max_queue: 200,
fallback_policy: :caller_runs
)
@pool ||= Concurrent::FixedThreadPool.new(@threads,
# max_queue: 0 means unlimited items in the queue. This is so we don't lose any
# work when shutting down.
max_queue: 0,
# fallback_policy is probably unnessary here but it won't hurt to set is explictly
fallback_policy: :caller_runs)
end

# Fetch all of the facets and load them into the :_from_biblio table
Expand Down
5 changes: 4 additions & 1 deletion lib/call_number_browse/term_fetcher.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ def initialize(page_size: 10_000, threads: 4)
# @return Concurrent::FixedThreadPool
def pool
@pool ||= Concurrent::FixedThreadPool.new(@threads,
max_queue: 200,
# max_queue: 0 means unlimited items in the queue. This is so we don't lose any
# work when shutting down.
max_queue: 0,
# fallback_policy is probably unnessary here but it won't hurt to set is explictly
fallback_policy: :caller_runs)
end

Expand Down

0 comments on commit c88b9c3

Please sign in to comment.