It failed to pull gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master by image-rs

[liudalibj]

It failed to pull gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master by image-rs

When I followed the document to run peer-pod nginx demo succeed. (I run the demo on x86 arch, and the kata-agent using image-rs(0.2.0) to pull/unpack docker images).

But when I try to replace the docker image from nginx to others(eg. gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master) I got a failed result, from the cloud-api-adaptor log it said something like this.

...
2022/11/22 14:37:24 [adaptor/proxy] CreateContainer: successfully pulled image "k8s.gcr.io/pause:3.7"
2022/11/22 14:37:24 [adaptor/proxy] StartContainer: containerID:a78e2f87afbc5c67cdc4bc08e48cdc9fdf6bd37134e31b3a8faadea9a94e0d30
2022/11/22 14:37:24 [adaptor/proxy] PullImage: image:gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master containerID:
2022/11/22 14:37:30 [adaptor/proxy] PullImage fails: rpc error: code = Internal desc = unpack destination "/run/image/layers/sha256_454a983f8cf90460aa762df71208b2d19e4db94bd97070bf111d872a86557fa2" already exists
2022
...

Login to the podVM and check related agent-protocol-forwarder/kata-agent log:

...
Nov 22 14:37:53 podvm-nginx-a78e2f87 agent-protocol-forwarder[653]: 2022/11/22 14:37:53 [forwarder/interceptor] PullImage: image: "gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master", containerID: ""
Nov 22 14:37:53 podvm-nginx-a78e2f87 kata-agent[695]: {"msg":"pull image \"ibm-vpc-block-csi-driver_master\", bundle path \"/run/kata-containers/ibm-vpc-block-csi-driver_master\"","level":"INFO","ts":"2022-11-22T14:37:53.243077922Z","version":"0.1.0","name":"kata-agent","subsystem":"rpc","pid":"695","source":"agent"}
Nov 22 14:37:55 podvm-nginx-a78e2f87 agent-protocol-forwarder[653]: 2022/11/22 14:37:55 [forwarder/interceptor] PullImage failed with error: rpc error: code = Internal desc = unpack destination "/run/image/layers/sha256_da8ef40b9ecabc2679fe2419957220c0272a965c5cf7e0269fa1aeeb8c56f2e1" already exists
...

This issue can be reproduced simply, after we got image-rs codes:

Add the test images to vet at here:
https://github.com/confidential-containers/image-rs/blob/main/src/image.rs#L351

        let oci_images = vec![
            "gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master",
            // Alibaba Container Registry
            "registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2",
            

And then run

cd ~/image-rs
RUST_BACKTRACE=full cargo test -p image-rs --lib -- image::tests::test_pull_image

in the console it will show something like:

root@peer-pod1:~/git/image-rs#  RUST_BACKTRACE=full cargo test -p image-rs --lib -- image::tests::test_pull_image
   Compiling image-rs v0.1.0 (/root/git/image-rs)
    Finished test [unoptimized + debuginfo] target(s) in 22.70s
     Running unittests src/lib.rs (target/debug/deps/image_rs-9395aaee605d7206)

running 1 test
test image::tests::test_pull_image ... FAILED

failures:

---- image::tests::test_pull_image stdout ----
thread 'image::tests::test_pull_image' panicked at 'assertion failed: image_client.pull_image(image, bundle_dir.path(), &None, &None).await.is_ok()', src/image.rs:372:13
stack backtrace:
   0:     0x5649bf99107d - std::backtrace_rs::backtrace::libunwind::trace::h9135f25bc195152c
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x5649bf99107d - std::backtrace_rs::backtrace::trace_unsynchronized::h015ee85be510df51
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5649bf99107d - std::sys_common::backtrace::_print_fmt::h5fad03caa9652a2c
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:66:5
   3:     0x5649bf99107d - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h2b42ca28d244e5c7
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:45:22
   4:     0x5649bf9b91ec - core::fmt::write::h401e827d053130ed
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/fmt/mod.rs:1198:17
   5:     0x5649bf98ab51 - std::io::Write::write_fmt::h1da442c9e1df41d6
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/io/mod.rs:1672:15
   6:     0x5649bf992b55 - std::sys_common::backtrace::_print::h180c4c706ee1d3fb
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:48:5
   7:     0x5649bf992b55 - std::sys_common::backtrace::print::hd0c35d18765761c9
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:35:9
   8:     0x5649bf992b55 - std::panicking::default_hook::{{closure}}::h1f023310983bc730
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:295:22
   9:     0x5649bf9927f4 - std::panicking::default_hook::h188fec3334afd5be
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:311:9
  10:     0x5649bf9930e6 - std::panicking::rust_panic_with_hook::hf26e9d4f97b40096
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:698:17
  11:     0x5649bf992f99 - std::panicking::begin_panic_handler::{{closure}}::hfab912107608087a
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:586:13
  12:     0x5649bf991574 - std::sys_common::backtrace::__rust_end_short_backtrace::h434b685ce8d9965b
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:138:18
  13:     0x5649bf992d09 - rust_begin_unwind
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:584:5
  14:     0x5649bdf2fb13 - core::panicking::panic_fmt::ha6dc7f2ab2479463
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:142:14
  15:     0x5649bdf2f9dd - core::panicking::panic::hb3ad04c589a0e3c8
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:48:5
  16:     0x5649be03a17b - image_rs::image::tests::test_pull_image::{{closure}}::haa95065799618c5b
                               at /root/git/image-rs/src/image.rs:372:13
  17:     0x5649be13e0ab - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h56bbb5dd8c21e3c7
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  18:     0x5649bdf3ba4f - <core::pin::Pin<P> as core::future::future::Future>::poll::h403005a610d5e603
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/future.rs:124:9
  19:     0x5649bdfd4a10 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}::h7d625bbd5e9f4a96
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:531:57
  20:     0x5649bdfd3151 - tokio::runtime::coop::with_budget::h23766a9f85bfbfd1
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/coop.rs:102:5
  21:     0x5649bdfd3151 - tokio::runtime::coop::budget::hc0e47a9ca91ae664
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/coop.rs:68:5
  22:     0x5649bdfd3151 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::h895963fc68d9c1be
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:531:25
  23:     0x5649bdfbe5f2 - tokio::runtime::scheduler::current_thread::Context::enter::hd317f6309a709d87
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:340:19
  24:     0x5649bdfd15ba - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::hfd118c1ea070e9a9
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:530:36
  25:     0x5649bdfc6aab - tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}::h4f37930b0f96bcc3
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:601:57
  26:     0x5649bdf90abd - tokio::macros::scoped_tls::ScopedKey<T>::set::hf684ef6521affd2e
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/macros/scoped_tls.rs:61:9
  27:     0x5649bdfc4fba - tokio::runtime::scheduler::current_thread::CoreGuard::enter::h93ea693d122bbe2c
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:601:27
  28:     0x5649bdfc7374 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::h8465de57b77c6cfc
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:520:19
  29:     0x5649bdfb4c7e - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::h54b2b7455e8f59fd
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/scheduler/current_thread.rs:154:24
  30:     0x5649bdfaaea0 - tokio::runtime::runtime::Runtime::block_on::h7b5a195dbcff15ed
                               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.22.0/src/runtime/runtime.rs:279:47
  31:     0x5649be14786b - image_rs::image::tests::test_pull_image::h2a44c1fc86a2fd8c
                               at /root/git/image-rs/src/image.rs:378:9
  32:     0x5649be039a2a - image_rs::image::tests::test_pull_image::{{closure}}::h11550357f7c1ac5c
                               at /root/git/image-rs/src/image.rs:343:11
  33:     0x5649be0fb43e - core::ops::function::FnOnce::call_once::h7e42e01d062a9575
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5
  34:     0x5649be21fba3 - core::ops::function::FnOnce::call_once::h2da14387562cd1a6
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5
  35:     0x5649be21fba3 - test::__rust_begin_short_backtrace::h307c198d88e9e51a
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/test/src/lib.rs:572:5
  36:     0x5649be21e9aa - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h42564b04a6d1df41
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/boxed.rs:1935:9
  37:     0x5649be21e9aa - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h286b0e8427b9af31
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panic/unwind_safe.rs:271:9
  38:     0x5649be21e9aa - std::panicking::try::do_call::h0b094c8aa8626e3b
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:492:40
  39:     0x5649be21e9aa - std::panicking::try::h253dd43de0385d2d
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:456:19
  40:     0x5649be21e9aa - std::panic::catch_unwind::h8a89f69958871bf1
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panic.rs:137:14
  41:     0x5649be21e9aa - test::run_test_in_process::h79875496f9bc15c6
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/test/src/lib.rs:595:18
  42:     0x5649be21e9aa - test::run_test::run_test_inner::{{closure}}::hf979093fbdf02646
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/test/src/lib.rs:489:39
  43:     0x5649be1eabdd - test::run_test::run_test_inner::{{closure}}::he01c2dd4903b57d1
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/test/src/lib.rs:516:37
  44:     0x5649be1eabdd - std::sys_common::backtrace::__rust_begin_short_backtrace::hece2f1c514466a0a
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:122:18
  45:     0x5649be1f0548 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h658be5979f99b3a5
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/mod.rs:505:17
  46:     0x5649be1f0548 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h9dbd94372ee944af
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panic/unwind_safe.rs:271:9
  47:     0x5649be1f0548 - std::panicking::try::do_call::hff71896f3757c3b2
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:492:40
  48:     0x5649be1f0548 - std::panicking::try::hbb4c2c2616b6a189
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:456:19
  49:     0x5649be1f0548 - std::panic::catch_unwind::h72f47810ab7e379e
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panic.rs:137:14
  50:     0x5649be1f0548 - std::thread::Builder::spawn_unchecked_::{{closure}}::h6331fb85a785812c
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/mod.rs:504:30
  51:     0x5649be1f0548 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h467656fda03180e5
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5
  52:     0x5649bf998bc3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h56d5fc072706762b
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/boxed.rs:1935:9
  53:     0x5649bf998bc3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h41deef8e33b824bb
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/boxed.rs:1935:9
  54:     0x5649bf998bc3 - std::sys::unix::thread::Thread::new::thread_start::ha6436304a1170bba
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys/unix/thread.rs:108:17
  55:     0x7f2b5f1a2609 - start_thread
  56:     0x7f2b5ef70133 - clone
  57:                0x0 - <unknown>


failures:
    image::tests::test_pull_image

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 53 filtered out; finished in 58.19s

error: test failed, to rerun pass '-p image-rs --lib'
root@peer-pod1:~/git/image-rs#

Here is the Dockerfile for gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master

[Xynnn007]

Hi @liudalibj , thanks a lot for reporting this bug and providing reproducing steps! This issue may be related to confidential-containers/enclave-cc#60 as well.

As I tested, the test image you've given gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master has two equal layers, whose hash is "sha256:454a983f8cf90460aa762df71208b2d19e4db94bd97070bf111d872a86557fa2"

The logic inside image-rs will add the information of the new layers after pulling the whole set of layers. However, the pulling logic will try to find whether there is any duplicated layer and if yes there will raise an error.

Thus for those images who have duplicated layers will all show this error.

I think there should add some code to filter duplicated OciDescriptor of the manifest which is pulled before pulling the layers. Is this rationale? @arronwy

[liudalibj]

@Xynnn007 @arronwy Thanks for your quick actions!