等待SuckerPunch::Job任务完成

Question

我尝试使用Sucker Punch gem来并行处理任务。但是我发现没有文档说明如何等待其终止。

require 'sucker_punch'

class SuckerJob
  include SuckerPunch::Job
  workers 4

  def perform(event)
    sleep(rand(5))
    puts "[#{Thread.current.object_id}] End processing event   #{event}."
  end
end

10.times { |i| SuckerJob.perform_async(i) }
puts "Shutting down ..."
SuckerPunch::Queue.shutdown_all
puts "Shutdown finished, status: #{SuckerPunch::Queue.stats[SuckerJob.name]}"
# [Ugly] call internal method
SuckerPunch::Queue::QUEUES.fetch_or_store(SuckerJob.name).wait_for_termination(10)
puts "Wait finished, status: #{SuckerPunch::Queue.stats[SuckerJob.name]}"

看起来，SuckerPunch::Queue.shutdown_all()在所有任务完成之前就返回了。

Shutting down ...
[17487240] End processing event   1.
[17488760] End processing event   0.
[17487240] End processing event   4.
[17488760] End processing event   5.
[17486120] End processing event   2.
[17484940] End processing event   3.
[17487240] End processing event   6.
Shutdown finished, status: {"workers"=>{"total"=>3, "busy"=>3, "idle"=>0}, "jobs"=>{"processed"=>7, "failed"=>0, "enqueued"=>0}}
[17484940] End processing event   9.
[17488760] End processing event   7.
[17486120] End processing event   8.
Wait finished, status: {"workers"=>{"total"=>0, "busy"=>0, "idle"=>0}, "jobs"=>{"processed"=>10, "failed"=>0, "enqueued"=>0}}

如何等待所有任务完成？

- sschmeck

---

你可以尝试使用 SuckerPunch::Counter::Processed.new(@queue).value。 - Rajarshi Das

要检查统计信息，您可以执行 all_stats = SuckerPunch::Queue.stats 然后 stats = all_stats[MyJob.to_s]，然后您现在可以查看 stats["jobs"]["processed"]，现在有 total、processed、busy 等。 - Rajarshi Das

Answer 1

您可以检查状态，

或者您可以检查统计数据。

all_stats = SuckerPunch::Queue.stats 

那么。

stats = all_stats[SuckerJob.to_s] 

那么现在您可以看到的统计数据。

stats["jobs"]["processed"]  

你可以像下面这样检查：

  stats["jobs"]["processed"] > 0
  stats["jobs"]["failed"] == 0
   stats["jobs"]["enqueued"] == 0

- Rajarshi Das

Answer 2

我使用这个：

def wait_for_jobs(job_name:, count:, max_seconds: 100)
  Rails.logger.info "Waiting up to #{max_seconds} seconds for #{count} jobs to run"
  wait_time = 0
  while wait_time < max_seconds
    stats = SuckerPunch::Queue.stats[job_name]
    processed = stats['jobs']['processed']
    break unless processed < count

    sleep(1)
    wait_time += 1
  end
  raise StandardError, "Timeout while waiting for #{count} jobs of #{job_name} to have run!" unless wait_time < max_seconds

  Rails.logger.info "#{count} jobs took #{wait_time} seconds to run"
end

- AlejandroVD