Using pip-sync in Google Colab (Jupyter Notebook)

[1951FDG]

With pip-sync packages will be uninstalled and re-installed, some of the packages are not available pypi, e.g., Google specific packages

For example, I would like for a selected group of packages to be untouched, left as is. How can I achieve that using pip-sync?

Environment Versions

1. Google Colab (Jupyter Notebook)
2. Python version: Python 3.10.12
3. pip version: pip 24.3.1
4. pip-tools version: 7.4.1

Steps to replicate

1. ...
2. ...
3. ...

Expected result

...

Actual result

...

[WhyNotHugo]

Packages that meet the ones specified in the requirements file are left untouched. Can you provide reproduction steps for an example scenario?

[1951FDG]

Hi,

I compiled a list of packages (deemed untouchable for colab env) for pip-sync to not touch:

gspread==6.1.4
gspread-dataframe==4.0.0
ipyevents==2.0.2
ipyfilechooser==0.6.0
ipykernel==5.5.6
ipyleaflet==0.19.2
ipyparallel==8.8.0
ipython==7.34.0
ipython-genutils==0.2.0
ipython-sql==0.5.0
ipytree==0.2.2
ipywidgets==7.7.1
pyzmq==24.0.1
tornado==6.3.3
google==2.0.3
google-ai-generativelanguage==0.6.10
google-api-core==2.19.2
google-api-python-client==2.155.0
google-auth==2.27.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.1
google-cloud-aiplatform==1.74.0
google-cloud-bigquery==3.25.0
google-cloud-bigquery-connection==1.17.0
google-cloud-bigquery-storage==2.27.0
google-cloud-bigtable==2.27.0
google-cloud-core==2.4.1
google-cloud-datastore==2.20.2
google-cloud-firestore==2.19.0
google-cloud-functions==1.19.0
google-cloud-iam==2.17.0
google-cloud-language==2.16.0
google-cloud-pubsub==2.27.2
google-cloud-resource-manager==1.14.0
google-cloud-storage==2.19.0
google-cloud-translate==3.19.0
google-colab==1.0.0
google-crc32c==1.6.0
google-genai==0.3.0
google-generativeai==0.8.3
google-pasta==0.2.0
google-resumable-media==2.7.2
googleapis-common-protos==1.66.0
googledrivedownloader==0.4
grpc-google-iam-v1==0.14.0
pydata-google-auth==1.9.0

I then passed this list (newline separated) to piptools, also please note that both requirements and dev_requirements already compiled successfully.

!echo "$extra_requirements" | python -m piptools compile --constraint=requirements.txt --constraint=dev_requirements.txt --output-file=extra_requirements.txt --pip-args='--progress-bar off -qq' -q --strip-extras -

I then get:

ERROR: Ignored the following versions that require a different python version: 2.10.0 Requires-Python >=3.6, <3.10; 2.11.0 Requires-Python >=3.6, <3.10; 2.12.0 Requires-Python >=3.6, <3.10; 2.13.0 Requires-Python >=3.6, <3.10; 2.13.1 Requires-Python >=3.6, <3.10; 2.14.0 Requires-Python >=3.6, <3.10; 2.15.0 Requires-Python >=3.6, <3.10; 2.16.0 Requires-Python >=3.6, <3.10; 2.16.1 Requires-Python >=3.6, <3.10; 2.17.0 Requires-Python >=3.6, <3.10; 2.18.0 Requires-Python >=3.6, <3.10; 2.19.0 Requires-Python >=3.6, <3.10; 2.20.0 Requires-Python >=3.6, <3.10; 2.21.0 Requires-Python >=3.6, <3.10; 2.22.0 Requires-Python >=3.6, <3.10; 2.22.1 Requires-Python >=3.6, <3.10; 2.23.0 Requires-Python >=3.6, <3.10; 2.23.1 Requires-Python >=3.6, <3.10; 2.23.2 Requires-Python >=3.6, <3.10; 2.23.3 Requires-Python >=3.6, <3.10; 2.24.0 Requires-Python >=3.6, <3.10; 2.24.1 Requires-Python >=3.6, <3.10; 2.25.0 Requires-Python >=3.6, <3.10; 2.25.1 Requires-Python >=3.6, <3.10; 2.25.2 Requires-Python >=3.6, <3.10; 2.26.0 Requires-Python >=3.6, <3.10; 2.27.0 Requires-Python >=3.6, <3.10; 2.27.1 Requires-Python >=3.6, <3.10; 2.28.0 Requires-Python >=3.6, <3.10; 2.28.1 Requires-Python >=3.6, <3.10; 2.29.0 Requires-Python >=3.6, <3.10; 2.6.2 Requires-Python >=3.6, <3.9; 2.7.0 Requires-Python >=3.6, <3.10; 2.8.0 Requires-Python >=3.6, <3.10; 2.9.0 Requires-Python >=3.6, <3.10

ERROR: Could not find a version that satisfies the requirement google-colab==1.0.0 (from versions: none)

[webknjaz]

Well, the depresolver still needs to validate the deps somehow to make sure that newly added packages don't violate the restrictions in the currently installed ones too. Perhaps, doing a pip install --no-deps would skip that, as a workaround.

Alternatively, you may want to consider archiving a wheelhouse and sending that into your env, I suppose.

[webknjaz]

Although, if those custom packages are available via an additional index, you should stick that into the command/config through custom pip args.

[1951FDG]

I believe google-colab is only available from https://anaconda.org/conda-forge/google-colab, but have no clue how to make that available via an additional index, do you by any chance?

[webknjaz]

If you're getting things from an entirely different ecosystem, regular dependency resolvers won't work. Also, it's usually a bad idea to try to make different dependency management tools control the same virtualenv — they don't know of each other and the metadata of the other side. You might need to be creative here.

FWIW, you don't have to use pip-sync specifically — I think pip install is able to skip the pre-satisfied packages. You should be able to enforce their versions via --constraint.