Embrace the frozen set #3

nineteendo · 2024-06-04T17:44:15Z

Frozen set literals and comprehensions

We should embrace the frozen set, literally: {{...}}.

Motivation

Currently you need too many characters to construct a frozen set:

foo = frozenset({1, 2, 3})
bar = frozenset({a, b, c})
baz = frozenset({c * 2 for c in "abc"})

Additionally, this is very inefficient:

>>> from dis import dis
>>> dis('foo = frozenset({1, 2, 3})')
  0           RESUME                   0

  1           LOAD_NAME                0 (frozenset)
              PUSH_NULL
              BUILD_SET                0
              LOAD_CONST               0 (frozenset({1, 2, 3}))
              SET_UPDATE               1
              CALL                     1
              STORE_NAME               1 (foo)
              RETURN_CONST             1 (None)
>>> dis('bar = frozenset({a, b, c})')
  0           RESUME                   0

  1           LOAD_NAME                0 (frozenset)
              PUSH_NULL
              LOAD_NAME                1 (a)
              LOAD_NAME                2 (b)
              LOAD_NAME                3 (c)
              BUILD_SET                3
              CALL                     1
              STORE_NAME               4 (bar)
              RETURN_CONST             0 (None)
>>> dis('baz = frozenset({c * 2 for c in "abc"})')
   0           RESUME                   0

   1           LOAD_NAME                0 (frozenset)
               PUSH_NULL
               LOAD_CONST               0 ('abc')
               GET_ITER
               LOAD_FAST_AND_CLEAR      0 (c)
               SWAP                     2
       L1:     BUILD_SET                0
               SWAP                     2
       L2:     FOR_ITER                 7 (to L3)
               STORE_FAST_LOAD_FAST     0 (c, c)
               LOAD_CONST               1 (2)
               BINARY_OP                5 (*)
               SET_ADD                  2
               JUMP_BACKWARD            9 (to L2)
       L3:     END_FOR
               POP_TOP
       L4:     SWAP                     2
               STORE_FAST               0 (c)
               CALL                     1
               STORE_NAME               1 (baz)
               RETURN_CONST             2 (None)

  --   L5:     SWAP                     2
               POP_TOP

   1           SWAP                     2
               STORE_FAST               0 (c)
               RERAISE                  0
ExceptionTable:
  L1 to L4 -> L5 [4]

That's why it could be useful to have frozen set literals and comprehensions:

foo = {{1, 2, 3}}
bar = {{a, b, c}}
baz = {{c * 2 for c in "abc"}}

Then this is all we need to do:

>>> from dis import dis
>>> dis('foo = {{1, 2, 3}}')
  0           RESUME                   0

  1           LOAD_CONST               0 (frozenset({1, 2, 3}))
              STORE_NAME               0 (foo)
              RETURN_CONST             1 (None)
>>> dis('bar = {{a, b, c}}')
  0           RESUME                   0

  1           LOAD_NAME                0 (a)
              LOAD_NAME                1 (b)
              LOAD_NAME                2 (c)
              BUILD_FROZENSET          3
              STORE_NAME               3 (bar)
              RETURN_CONST             0 (None)
>>> dis('baz = {{c * 2 for c in "abc"}}')
   0           RESUME                   0

   1           LOAD_CONST               0 ('abc')
               GET_ITER
               LOAD_FAST_AND_CLEAR      0 (c)
               SWAP                     2
       L1:     BUILD_FROZENSET          0
               SWAP                     2
       L2:     FOR_ITER                 7 (to L3)
               STORE_FAST_LOAD_FAST     0 (c, c)
               LOAD_CONST               1 (2)
               BINARY_OP                5 (*)
               SET_ADD                  2
               JUMP_BACKWARD            9 (to L2)
       L3:     END_FOR
               POP_TOP
       L4:     SWAP                     2
               STORE_FAST               0 (c)
               STORE_NAME               0 (baz)
               RETURN_CONST             2 (None)

  --   L5:     SWAP                     2
               POP_TOP

   1           SWAP                     2
               STORE_FAST               0 (c)
               RERAISE                  0
ExceptionTable:
  L1 to L4 -> L5 [2]

Syntax

frozenset_display ::= "{{" (`starred_list` | `comprehension`) "}}"

Note

Technically the syntax is this:

frozenset_display ::= "{{" (`starred_list` | `comprehension`) "}" "}"

But that's an implementation detail that shouldn't be documented.

Examples

Example 1

assert {{1, 2, 3}}         == frozenset({1, 2, 3})
assert { {{1, 2, 3}} }     == {frozenset({1, 2, 3})}
assert {{{{1, 2, 3}}}}     == frozenset({frozenset({1, 2, 3})})
assert { {{{{1, 2, 3}}}} } == {frozenset({frozenset({1, 2, 3})})}
...

Note

These are still type errors:

foo = { {1, 2, 3} }     # TypeError: unhashable type: 'set'
bar = {{{1, 2, 3}}}     # TypeError: unhashable type: 'set'
baz = {{ {1, 2, 3} }}   # TypeError: unhashable type: 'set'
qux = { { {1, 2, 3} } } # TypeError: unhashable type: 'set'
...

And these are now syntax errors:

foo = {{}}
bar = {{"a": 1, "b": 2, "c": 3}}

Example 2

assert {{c * 2 for c in "abc"}} == frozenset({c * 2 for c in "abc"})

Note

This is still a type error:

foo = { {c * 2 for c in "abc"} } # TypeError: unhashable type: 'set'

And this is now a syntax error:

foo = {{c: c * 2 for c in "abc"}}

Backwards compatibility

These statements would behave differently with this proposal:

foo = {{1, 2, 3}}          # TypeError: unhashable type: 'set'
bar = { {{1, 2, 3}} }      # TypeError: unhashable type: 'set'
baz = {{{{1, 2, 3}}}}      # TypeError: unhashable type: 'set'
qux = { {{{{1, 2, 3}}}} }  # TypeError: unhashable type: 'set'
...
foo = { {c * 2 for c in "abc"} } # TypeError: unhashable type: 'set'

But I don't think they can be used for anything useful, -'' is a shorter way to raise a type error.

Pros

Symmetrical and only uses brackets
Doesn't raise syntax error in previous versions
Doesn't reserve any new syntax
Double braces can look intuitively as a hardened set
Some other languages already give special meaning to {{...}}
Some text editors already support embracing selected text

Cons

Potentially breaks backwards compatibility
~~Double punctuation isn't Pythonic~~, there's a precedent with triple quotes: '''''+'''''
Suffers from brace overflow
Requires a lot of changes to tokenisation and string representations

GitHub usage

>28.8k files with frozenset({...})

Other suggestions for frozenset literals

expand

`{1, 2, 3}.freeze()`

Example:

assert {1, 2, 3}.freeze() == frozenset({1, 2, 3})

Pros:

Intuitive
Doesn't raise syntax error in previous versions

Cons:

Hard to maintain
Unclear that it wouldn't be copied at runtime
Doesn't improve representation

`{1, 2, 3}`

Example:

assert {1, 2, 3} == frozenset({1, 2, 3})
assert {1, 2, 3} != set({1, 2, 3})

Pros:

Symmetrical and only uses brackets

Cons:

Not backwards compatible
Inconsistent way to get mutable collections
No unambiguous notation for set literals

`|1, 2, 3|`

Example:

assert |1, 2, 3| == frozenset({1, 2, 3})

Cons:

Undirectional
Can't keep track of nesting

`<1, 2, 3>`

Example:

assert <1, 2, 3> == frozenset({1, 2, 3})

Pros:

Symmetrical and only uses brackets

Cons:

Can't keep track of nesting
Hard to read
Already used as operator

`f{1, 2, 3}`

Example:

assert f{1, 2, 3} == frozenset({1, 2, 3})

Pros:

Prefix can be easily added and removed

Cons:

Rules out possibility of foo{...}
Not obvious
Looks like slicing operator or function call
Endless arguing over s{} for empty set

`{{1, 2, 3}}`

Example:

assert {{1, 2, 3}}       == frozenset({1, 2, 3})
assert {{{1, 2, 3}}}     == {frozenset({1, 2, 3})}
assert {{{{1, 2, 3}}}}   == frozenset({frozenset({1, 2, 3})})
assert {{{{{1, 2, 3}}}}} == {frozenset({frozenset({1, 2, 3})})}
...

Pros:

Symmetrical and only uses brackets
Doesn't raise syntax error in previous versions
Doesn't reserve any new syntax
Double braces can look intuitively as a hardened set
Some other languages already give special meaning to {{...}}
Some text editors already support embracing selected text

Cons:

Potentially breaks backwards compatibility
Hard to read and parse
~~Double punctuation isn't Pythonic~~, there's a precedent with triple quotes: '''''+'''''
Suffers from brace overflow
Requires a lot of changes to tokenisation and string representations

`|{1, 2, 3}|`

Example:

assert |{1, 2, 3}| == frozenset({1, 2, 3})

Cons:

PEP 351 was rejected
Undirectional

Links

📚 Documentation preview 📚: https://cpython-previews--3.org.readthedocs.build/

nineteendo · 2024-06-05T09:39:08Z

@blhsing, I managed to implement frozen set literals. If the syntax needs to be modified that's very easy, all other code is already in place. I'll discuss this in 2025 on Discourse.

blhsing · 2024-06-05T10:17:38Z

@blhsing, I managed to implement frozen set literals. If the syntax needs to be modified that's very easy, all other code is already in place. I'll discuss this in 2025 on Discourse.

Great work. Looks very good to me so far. Will try to poke holes in it later.

nineteendo · 2024-06-12T14:22:28Z

I made it symmetrical: { {{1}}: 1 } I didn't like how it looked before. It also simplified the implementation.

nineteendo · 2024-06-12T15:06:41Z

Should we use frozenset literals in the representation of subclasses? Using them is faster than regular sets.

>>> class Set(set): pass
... 
>>> Set({1, 2, 3})
Set({1, 2, 3}) # use Set({{1, 2, 3}}) instead

It also simplifies the logic a bit.

This reverts commit ae28aff.

nineteendo · 2024-06-12T18:37:00Z

I've used the new syntax throughout the standard library (excluding tests). That made me find a library I forgot to update.

nineteendo · 2024-06-12T19:23:42Z

By chainging the format of frozenset's repr we would break tons of existing unit tests and decrease the chances of the proposal getting accepted. We should best roll back all my changes then.

The pull request is already way too large to review, so I'll have to split it up anyway. But this way we can show what the Python codebase could look like in a couple years and I'll have less work to figure things out if we decide to implement it anyway. We might be able to do this with a __future__ import though.

I had no idea { {} } is allowed in a type hint, and could not find any documentation about it even after you pointed it out. Can you show me where this behavior is documented, and what it means to have { {} } as a type hint (I see that it is somehow evaluated as a string rather than a set)?

It's not, but I didn't like the fact that this would raise a syntax error when evaluated and that { {1} } would result in a frozen set instead of a type error.

Could you ask for some initial feedback on Discourse in the help category? (My account was suspended)

blhsing · 2024-06-13T01:50:21Z

By chainging the format of frozenset's repr we would break tons of existing unit tests and decrease the chances of the proposal getting accepted. We should best roll back all my changes then.

The pull request is already way too large to review, so I'll have to split it up anyway. But this way we can show what the Python codebase could look like in a couple years and I'll have less work to figure things out if we decide to implement it anyway. We might be able to do this with a __future__ import though.

Fair enough. Perhaps the old repr output can go through a deprecation process to give people time to update their codebase.

I had no idea { {} } is allowed in a type hint, and could not find any documentation about it even after you pointed it out. Can you show me where this behavior is documented, and what it means to have { {} } as a type hint (I see that it is somehow evaluated as a string rather than a set)?

It's not, but I didn't like the fact that this would raise a syntax error when evaluated and that { {1} } would result in a frozen set instead of a type error.

Could you ask for some initial feedback on Discourse in the help category? (My account was suspended)

Ahh I've just found where this behavior is documented. It's called postponed evaluation of annotations as documented in PEP 563, where with from __future__ import annotations all annotations become strings at runtime as long as they're syntactically valid.

Sorry to hear about your suspended account by the way. I was wondering why you have to wait until 2025 to present this idea but I suppose that's when your account can be active again.

blhsing · 2024-06-13T01:56:50Z

Should we use frozenset literals in the representation of subclasses? Using them is faster than regular sets.
>>> class Set(set): pass
... 
>>> Set({1, 2, 3})
Set({1, 2, 3}) # use Set({{1, 2, 3}}) instead
It also simplifies the logic a bit.

That makes total sense. Still slightly annoyed that frozenset literals have to cost 2 more characters but the benefit in simplified bytecode is clearly there.

blhsing · 2024-06-13T01:59:47Z

I've used the new syntax throughout the standard library (excluding tests). That made me find a library I forgot to update.

Great job finding usages in the stdlib, signifying the usefulness of the change.

nineteendo · 2024-06-13T06:18:03Z

Sorry to hear about your suspended account by the way. I was wondering why you have to wait until 2025 to present this idea but I suppose that's when your account can be active again.

My account is suspended until September, but because I made too many half baked suggestions I was asked to wait until 2025.

nineteendo · 2024-07-14T14:58:20Z

@Gouvernathor, would it make sense to already think about frozendict literals? Because changing the repr in the next version would be a breaking change. Most of what I did in this proof of concept could be adjusted for frozen dict literals. And the PEP could be called: "Embrace the frozen dict" (literally).

It might also be better to add frozen dict literals first. Otherwise people will ask for {{}} to create an empty frozen set.

Gouvernathor · 2024-07-14T18:33:58Z

I'm not sure how I feel about this proposition in itself. I'd be intuitively in favor of a f{} syntax - but if the consensus is this way, I can get behind it.
Aside that, since {} is already taken by dicts, then it makes no sense to use {{}} for frozen sets, even if frozen dicts never happen it just would make the language inconsistent. And you can indeed argue that frozen dicts may be coming, in order to "reserve" the empty double-brace syntax.

nineteendo · 2024-07-14T19:26:01Z

I'm not sure how I feel about this proposition in itself. I'd be intuitively in favor of a f{} syntax - but if the consensus is this way, I can get behind it.

Opinions vary, some find it intuitive, others find it very UNintuitive. From what I have gathered there are simply less issues with {{...}} (see Other suggestions for frozenset literals). I'm planning to do a poll in January.

nineteendo added 28 commits June 4, 2024 18:44

Add double lbrace token

1e798bb

Construct frozenset

e104d43

Construct dynamic frozensets

6f807cb

Add grammar tests

366ab41

Document new node

a5aeefe

Fix grammar

fc48a5c

Document CALL_INTRINSIC_1

e714040

Test dis

3bbf27f

Fix indent in unparsed ast

32d83c0

Add extra test

a130604

Test node

f10508f

Update stdtypes

0ecfd57

Build frozenset opcode

63096f4

Update expected ops

0719566

Mention frozen sets in data structures

c433fd3

Explain output format

c1f07cf

Clarify {{}}

1aebc4c

Document grammar

aabe7b8

Fix grammar

b8a6142

Revert unneeded change

c51632b

Add frozen

8f2af14

Add to atoms

56057fa

Rename variable

580eb6a

Rename variable 2

ee5eac8

Add note in grammar file

42093f0

Make note shorter

fed894a

Add nested test

0ad89fc

Test in dict

7373969

nineteendo added 3 commits June 12, 2024 12:03

Fix test name

534d418

Test with set of frozensets

a97267b

Test with set of frozensets

6cff396

nineteendo changed the title ~~Frozen set literals and comprehensions~~ Embrace the frozen set Jun 12, 2024

nineteendo added 4 commits June 12, 2024 16:00

Make symmetric

2e9d223

Use list for items

cf09b14

Document frozensets in dicts and sets

f8cd830

Add empty line

47e042a

nineteendo added 5 commits June 12, 2024 16:28

Add example with comprehension

dd2a13b

Handle brackets in frozenset subclasses

488b614

Fix subclass formatting

65ac5a0

Allign

6b5d9c4

Add more tests

ad67234

nineteendo added 5 commits June 12, 2024 17:21

Use frozenset in subclass representation

1ef5255

Use new syntax in stdlib

e7e6601

Use frozenset literals in misc files

ae28aff

Revert "Use frozenset literals in misc files"

c97028b

This reverts commit ae28aff.

Merge branch 'main' into frozenset-literals

2097356

nineteendo mentioned this pull request Jun 28, 2024

Support tuples for [r]find() & [r]index() #2

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Embrace the frozen set #3

Embrace the frozen set #3

nineteendo commented Jun 4, 2024 •

edited

Loading

nineteendo commented Jun 5, 2024 •

edited

Loading

blhsing commented Jun 5, 2024

nineteendo commented Jun 12, 2024 •

edited

Loading

nineteendo commented Jun 12, 2024

nineteendo commented Jun 12, 2024 •

edited

Loading

nineteendo commented Jun 12, 2024 •

edited

Loading

blhsing commented Jun 13, 2024 •

edited

Loading

blhsing commented Jun 13, 2024 •

edited

Loading

blhsing commented Jun 13, 2024 •

edited

Loading

nineteendo commented Jun 13, 2024

nineteendo commented Jul 14, 2024 •

edited

Loading

Gouvernathor commented Jul 14, 2024

nineteendo commented Jul 14, 2024

Embrace the frozen set #3

Are you sure you want to change the base?

Embrace the frozen set #3

Conversation

nineteendo commented Jun 4, 2024 • edited Loading

Frozen set literals and comprehensions

Motivation

Syntax

Examples

Example 1

Example 2

Backwards compatibility

Pros

Cons

GitHub usage

Other suggestions for frozenset literals

{1, 2, 3}.freeze()

{1, 2, 3}

|1, 2, 3|

<1, 2, 3>

f{1, 2, 3}

{{1, 2, 3}}

|{1, 2, 3}|

Links

nineteendo commented Jun 5, 2024 • edited Loading

blhsing commented Jun 5, 2024

nineteendo commented Jun 12, 2024 • edited Loading

nineteendo commented Jun 12, 2024

nineteendo commented Jun 12, 2024 • edited Loading

nineteendo commented Jun 12, 2024 • edited Loading

blhsing commented Jun 13, 2024 • edited Loading

blhsing commented Jun 13, 2024 • edited Loading

blhsing commented Jun 13, 2024 • edited Loading

nineteendo commented Jun 13, 2024

nineteendo commented Jul 14, 2024 • edited Loading

Gouvernathor commented Jul 14, 2024

nineteendo commented Jul 14, 2024

nineteendo commented Jun 4, 2024 •

edited

Loading

`{1, 2, 3}.freeze()`

`{1, 2, 3}`

`|1, 2, 3|`

`<1, 2, 3>`

`f{1, 2, 3}`

`{{1, 2, 3}}`

`|{1, 2, 3}|`

nineteendo commented Jun 5, 2024 •

edited

Loading

nineteendo commented Jun 12, 2024 •

edited

Loading

nineteendo commented Jun 12, 2024 •

edited

Loading

nineteendo commented Jun 12, 2024 •

edited

Loading

blhsing commented Jun 13, 2024 •

edited

Loading

blhsing commented Jun 13, 2024 •

edited

Loading

blhsing commented Jun 13, 2024 •

edited

Loading

nineteendo commented Jul 14, 2024 •

edited

Loading