Enable conditional_put for s3 by default

[ion-elgreco]

This is something we did in delta-rs, since all known S3 stores have this support now.

Let me know what you think @kylebarron

[kylebarron]

Can you link to the delta-rs PR? I'm not the most knowledgeable about conditional puts and I don't know what would change.

[ion-elgreco]

delta-io/delta-rs#3091

Enabling it means, you can do a put request with PutMode create without it throwing errors of not supported file operation.

[kylebarron]

I glanced through that PR but still don't know what would change in obstore. This is what I currently expose in the config: https://developmentseed.org/obstore/latest/api/store/aws/#obstore.store.S3Config.conditional_put

so to enable conditional put currently you'd use

S3Store(..., conditional_put="etag")

is that correct?

[kylebarron]

Is there upstream discussion about making that the default behavior for AmazonS3?

[ion-elgreco]

I glanced through that PR but still don't know what would change in obstore. This is what I currently expose in the config: https://developmentseed.org/obstore/latest/api/store/aws/#obstore.store.S3Config.conditional_put

so to enable conditional put currently you'd use

S3Store(..., conditional_put="etag")
is that correct?

Correct, we opted to insert these flags by default.

Is there upstream discussion about making that the default behavior for AmazonS3?

I think in the original PR when support was added for this, they left it off by default since it wasn't fully supported, but to my understanding this fully support by AWS was added couple months later? I think we should check with @tustvold

[kylebarron]

I'm inclined to wait and follow whatever defaults object_store chooses. object_store v0.12 is expected to be released this month, right? Would you like to create an upstream issue asking/confirming the status of the default behavior of conditional put?

ref apache/arrow-rs#7080

[kylebarron]

https://github.com/apache/arrow-rs/blob/1019f5b27d3596077bcdd7e10b67e2c6d4cfbf02/object_store/src/aws/precondition.rs#L133-L138 this docstring should probably be updated to include native S3 as a store that supports it?

Maybe we should discuss assigning #[default] on the ETagMatch variant?