The When of Python

Grant Paton-Simpson – 2degrees
Benjamin Denham – AUT PhD Candidate, DataMasque

The Promise of Python:

What is Readability?

Armed with basic language knowledge
→ rapidly understand what code will do

However, ever-expanding language features...

• Provide more than "one obvious way"
• Make Python too big for your brain
• Hurt readability!

namedtuple()

vs

NamedTuple

vs

@dataclass

concurrent.futures

vs

asyncio

vs

threading / multiprocessing

Does it matter?

Having choices is useful!

Storytime: X-Wing Miniatures Game

The game kept getting better!

Fast-forward a few years...

35× as much
to learn!

"...it got more complicated, making it less easy to jump into the proverbial cockpit."

But how easy is it to learn modern Python?

• In order to start reading Python, you must understand all commonly used features.
• After a 1-day course, how comfortable would you be reading Python from 10 years ago vs today?
   Type-hinting...
    F-strings...
     Walrus operator (:=)...
      Positional-only parameters...
       Structural pattern matching (match)...
        ...

Not everyone has the time to learn that much Python

• Scientists
• School Teachers
• Data Analysts

Language creep threatens Python's popularity!

So should we just stop extending Python?

• Python must still adapt to survive and improve!
• But it's really hard to remove old features
• We need a way to shrink Python

Can we shrink Python?

• Languages can deprecate features and APIs
  • JavaScript, Java, C++, PHP
• Python only deprecated 4 modules since 3.0 (2008) – PEP 4

Python: The Good Parts?

Not enough room for all the good parts

What parts of Python would you take with you?

Everyday Python is a...

Python Constrictor!

So let's constrict Python!

The When of Python

The When of Python

(This is how we think Python should be used,
not how it is currently used)

String Formatting

(See: realpython.com/python-string-formatting)

String Formatting

'...'.format(**kwargs)

Data-Storage Objects

Data-Storage Objects

Structural Pattern Matching: The Promise

          match json_shape:
              case {'type': 'circle', 'radius': radius}:
                  return Circle(radius)
              case {'type': 'rectangle', 'dimensions': [width, height]}:
                  return Rectangle(width, height)
              case _:
                  raise ValueError('Not a shape')
          

• Python finally gets a concise switch statement!
• Elegantly unpack nested data structures
• Stealing another handy feature from functional languages

Structural Pattern Matching: The Reality

• A new mini-language within Python
• More subtleties than constructs like if-else:

  
                match value:
                    case str():
                        print(f'{value} is a string!')
                # Without parens after str...
                match value:
                    case str:
                        print('Oops, this case always matches and redefines str!')
                

• Bugs are hard to see if you're not confident in reading its syntax

Structural Pattern Matching

Concurrency

threading / multiprocessing:

          def download_post(post_num):
             url = f'https://jsonplaceholder.typicode.com/posts/{post_num}'
             return requests.get(url).json()

          posts = {}
          post_nums = range(5)
          threads = [
             threading.Thread(
                # Make post_num an explicit argument to avoid late-binding
                target=lambda post_num: setitem(posts, post_num, download_post(post_num)),
                kwargs={'post_num': post_num},
             ) for post_num in post_nums
          ]
          # Start all the threads
          for thread in threads:
             thread.start()
          # Wait for all threads to finish
          for thread in threads:
             thread.join()
          # Order by post_num
          print([posts[post_num] for post_num in post_nums])
          

Concurrency: concurrent.futures

Simpler interface for both
threading and multiprocessing:

           def download_post(post_num):
              url = f'https://jsonplaceholder.typicode.com/posts/{post_num}'
              return requests.get(url).json()

           with concurrent.futures.ThreadPoolExecutor() as executor:
              posts = executor.map(download_post, range(5))
              print(list(posts))
          

Easy to refactor existing code:
Practical Python Async for Dummies

Concurrency: asyncio

An alternative paradigm for single-process concurrency that avoids overheads of threads:

          async def download_post(session, post_num):
             url = f'https://jsonplaceholder.typicode.com/posts/{post_num}'
             async with session.get(url) as response:
                return await response.json()

          async def main():
             async with aiohttp.ClientSession() as session:
                return await asyncio.gather(*[
                   download_post(session, post_num) for post_num in range(5)
                ])

          print(asyncio.run(main()))
          

A step backwards for readability and learnability?

Concurrency: asyncio

The learning curve on [async] is enormous
...
Many, many tools in Python will get a non-async version and a async version and it might in the end double the size of the language
...
I think async is the future; threading is so hard to get right [and] so expensive.

― Raymond Hettinger, PyBay 2017

Concurrency

for else

                
                for animal in animals:
                    if animal == my_pet:
                        break
                else:
                    print('No pet found :(')
                
              

                
                for animal in animals:
                    if animal == my_pet:
                        break
                else:
                    print('Pet found :)')
                
              

              
              found_pet = False
              for animal in animals:
                  if animal == my_pet:
                      found_pet = True
                      break
              if not found_pet:
                  print('No pet found :(')
              
            

for else

Misc

The When of Python

How we made those decisions

1. Cull alternatives (create "one obvious way")
   • Keep the simple
   • Prefer versatility
   • Remove the unsafe
2. Shrink Python (define Everyday Python)
   • Prioritise common programming needs
   • Prefer easy to learn/remember

Out with the old, in with the new?

• Often new approaches obsolete the old:
  • f-strings
  • Data Classes
• But sometimes extensions add complexity:
  • asyncio
• Or come with risks if not used very carefully:
  • Structural Pattern Matching

Where to from here?

→ Wide collaboration
  → Consensus
    → Standardisation

How about a PEP?

Python has a strong history of using constructs to:

• Reduce disagreements
• Not leave convention to chance

The When of Python should be the next Python construct

Why not just leave it to convention?

• Python is no longer a single, cohesive community
  • Conventions are slower to spread organically
  • Conventions may diverge, creating confusion

Explicit guidance is better than implicit convention

What are current Python conventions?

Python Feature Usage

What's using so much asyncio?

What's using so much asyncio?

A When of Python could help fix these stats

Monitoring how a language is used could be a game-changer for language design

• Understand conventions – don't just ask, look as well
• Spot splintering to target community outreach
• Measure effectiveness of language guidance

There currently seems to be no such coordinated monitoring of Python usage

Asking "When?" makes language decisions easier

Language features can be added for niche uses without expanding Everyday Python

When or No

Where else to ask "When?"

• Project/team style guides
• Teaching material
• Blog posts

Why stop at language features?

• When to use libraries/frameworks?
• When to test?
• When to document?

Takeaways

• Python isn't as simple as it used to be
• Language creep could cost Python its popularity
• We need to constrict Everyday Python without giving up useful features
• Adopting a When of Python would simplify Python for beginners and experienced developers alike
• Asking "When or No" could even make language decisions easier

Thanks for Listening

Thoughts? Questions? Opinions?

Post-Conference Refinements

We had lots of great conversations and feedback at Kiwi Pycon 2022 that led us to refine the structure of the When of Python

The following slides present these new ideas

The When of Python

Common Python

• The Python features every coder writing or reading Python should understand
• Especially useful for teachers and anyone learning Python

Situational Python

• The Python features that are useful
  but not for everyone or not all the time
• It depends on the situation, for example:
  • Web development might need very different features than scientific Python, e.g. asyncio
  • Advanced library code might need advanced features, e.g. low-level threading / multi-processing

Everyday Python is Personal

• Your Everday Python is Common Python plus whatever you need from Situational Python in your application domain
• A web developer making high-concurrency applications will have a different Everyday Python than an astronomer

Deprecated Python

• The Python features we should avoid using wherever possible
• Existing better alternatives should be recommended