Expert Product Development with Python

Expert Product Development with Python is a hands-on programming course aimed at software, hardware, support engineers, or anyone experienced in using Python code, who wants to create or turn an existing Python code base into a software product.

Typical applications include: command line and tools flows, for hardware verification, for software tests, for data science and machine learning, or for running on embedded devices.

This course does not assume any prior experience in productizing software. Of course, it won’t hurt if you do.

• Level up your Python skills and learn advanced Python programming constructs and techniques
• Tools, techniques, and best practices to organize, debug, test, profile, package, distribute and maintain a good quality of your code base during your product lifecycle
• How to design Command Line Interfaces (CLIs)
• How to scale-up your application in Python using concurrency modules and C-Extensions
• How to integrate native resources and build products for different Python versions, Operating Systems and hardware architecture platforms
• How to distribute your product to the world using the Python Package Index (PyPI) server and other means.

To maximize the training value, prior attendance of the Doulos course Essential Python (or equivalent experience) is required.

It is assumed that the participants are, or have been, programming with Python on a regular basis and have the relevant experience with the topics taught in Essential Python.

Doulos training materials are renowned for being the most comprehensive and user friendly available. Their style, content and coverage is unique, and has made them sought after resources in their own right. The materials include:

• Fully indexed class notes creating a complete reference manual
• Workbook full of practical examples and solutions to help you apply your knowledge

Introduction

Introducing Product Development with Python
From organic grown scripts to product • Training Goal • Course Roadmap • Modules Overview

An Essential Tour of Python

An advanced tour of Python's essential features
Name Binding • Compiled or Interpreted? • Global vs Local • LEGB Scoping Rules • Argument Passing • Mutability • Function Arguments • Special Parameters • Object in Python • First Class Objects • Decorator • Lambda, Iterable and Comprehension •  F-strings • Virtual Environment

Debugging

Efficient debugging tools and techniques in Python
Builtins Introspection • inspect • inspect stack frame • debugging playground • starting pdb • pdb commands • interactive session • Breakpoints • Inspect and modify • pdb Cheat Sheet • Switching debugger • Visual debugger • The Scientific Method • Going Further

Pythonic OOP

Advanced Pythonic object-oriented programming techniques
Class 101 • Class methods • Alternative Constructors • Static Methods •Virtual methods • Private Name Mangling • Instance variables vs getter/setter • property • property setter and deleter • Interface and ABC • ABC in Python • ABC and collections • Multiple Inheritance • MRO • Multiple Inheritance and Interfaces • Mixins • __slots__• Bonus Annex

Modules and Package

A deeper understanding of modules and packages and how to use them
import • import under the hood • module caching • from…import •   import *   •   __name__   •   packages   •   import from Package • Running modules from the command line   • logical namespace   •   practical considerations.

Code as Docs

Ways to document and structure code for peers and code users
Documentation strings • documenting API • type hints • mypy • code formatting • black • isort

Distribution Package

How to create a distribution package
introduction • Really Simple Script (rss) • toward productization • Python Project Layout • pyproject.toml • Configuring Metadata • Distribution Package Metadata • classifiers • package dependencies • Controlling Package Contents • Prod/Dev Environment • Configuring Black and Isort • Python Application • Testing your Distribution Package • Install for Source • Building  • Going Further  • Bonus Annex

Testing

Introducing pytest and common tools and techniques used for testing python products
Introduction • Testing Considerations • Testing Strategy • The Four-Phase Test pattern • Fake and Test Doubles  • Doctest  for e2e Testing • Doctest and docstring • Pytest Architecture • A Simple Test • A Failing Test • Test Discovery • Grouping Tests in a Class • Testing that an Exception is Raised • Skipped Tests and Expected Failures • Parametrization •  Test Summary Report • Dropping into pdb • Test Fixtures • User-Defined Test Fixtures  •  Mocking •  Monkeypatch Fixtures  • Monkeypatch and Mocking Fixtures • pytest Good Practices

Logging

The Python logging module
Logging: An overview • Loggers • Log level • Handlers • Formatters • Loggers hierarchy • Filters • Logging Workflow • Practical Consideration • Config by Coding • Logging on an Embedded System • Additional Resources • Bonus Annex

Command Line Interface

How to create a sound command line interface in Python
Introduction • CLI Design Guidelines • Exiting Python Programs •Handling Signal • Custom Signal Handler • Dealing with Ctrl-C • Parsing Argument • Positional Arguments • Usage and Help • Option Arguments • Options With and Without Values • Optional Values • nargs • Description, Epilog, Help • Prefix, Choices, Required • Click • Command Examples • From argparse to click • Click Path and File • Command Group • Invoking Sub-commands• Improving User Experience • Tqdm Progress Bar • Rich

Quality Assurance

Common tools and techniques to assess and improve the quality of your python product
Static Analysis and Type Checkers • flake8 • pylint • Security Auditing • Practical Considerations • Integrating Static Analysis Tools • Type checkers • Code Coverage • Coverage Report • Coverage Area • Branch Coverage • Mutation Testing • mutmut Output • Analysing mutmut Report • Tips and Tricks

Concurrency

An overview of concurrency and available modules from the standard library
What is concurrency • CPU vs I/O bound tasks • Concurrent Programming Models • Concurrency in Python • Concurrency Modules • Modules Summary

Threading

Introducing the threading module to implement concurrency using OS-threads

Introduction • Thread Creation • Thread Creation Using Class • Thread and Program Termination • Joining Thread • Thread Local Data • Race Condition • Synchronizations Objects • Lock Object • Event Object • Condition Object (waiters) • Condition Object (notify) • Communicating between Threads • Working with the GIL • Bonus annex

Multiprocessing

Introducing the multiprocessing module to implement concurrency using OS processes
Introduction • Starting a New Process • Start Methods • Choosing a Start Method • Inheriting Resources • Join, Terminate and Exit Code • Inter-Process Communication • Pipe • Queues • Shared Memory • Manager • Process Pool • Task Offload Methods • Bonus Annex

Executors and Futures

Some of the high-level constructs provided by the concurrent.futures module
Introduction • Executor Object • Thread Executor Pool • Process Executor Pool • Future • Future and Deadlocks • Customizing Future Execution

AsyncIO

Introducing the asyncio module to implement concurrency using Python coroutines
Introduction • Async IO architecture • async, await and coroutines • asyncio.run() Error • asyncio Tasks  • Awaitable Objects  • Cancellation and Shielding  • Await on Multiple Objects  • await with Timeout  • Monitor Completion  • await for the Earliest Result  • Synchronization Objects  • Queues  • Subprocess  • Interacting with Subprocess  • Many Other Features

Interfacing Native Resources

How to interface native resources in Python
Introduction •  Interfacing Native resources  • Using the Standard library  • ctypes  •  numba  •  Embed C with CFFI  •  Building and Running CFFI   •   Compiling from C Header and Source Files • Building from a Shared Object File • Pointers and Structures • CFFI Build Script • ffi.new • cdef() – practical considerations •   To distribute CFFI with PyInstaller  •  Cython • Cython Language • Primes in Python • Primes in Cython • Compile with Cython • Comparing Numba, Cython, and cffi • Compare the Speed • Summary

Python Wheels

Deep dive into how to build Python wheels to integrate binary extension modules
Introduction • Package with Native Resources • Python Wheels • Pure vs Binary Wheels • Building binary wheels • Packaging Cython Extension • Packaging CFFI Extension • Cross-Distribution Linux Wheel • Manylinux • Manylinux Docker Images • Build a manylinux Wheel Manually • Supporting Multiple Platforms • Cross-Building Wheels • cibuildwheels

Profiling

Common python tools and techniques to spot performance bottlenecks
Introduction • Profilers Technology • Top-Down Strategy • Timing Execution • Benchmark • cProfile  • Sort by cumtime  • Visualizing Report  • Line profiling  • Gathering Line Profiling Data • Line Profiling Result •Profiling Memory Usage  • Line-by-Line Memory Profiling  • Memory Profiling Tips & Tricks  • Scalene  • Scalene CLI report  • Scalene CPU usage  • Scalene Memory Usage  • Optimization Loop  • Micro-benchmarking  • Choosing your Profilers • Bonus Annex

Shipping Your Product

How to ship a Python Project
Overview • Distribution Package Lifecycle • PyInstaller • Dist Folder • PyInstaller Option • Source Distribution • Publishing Locally • Publishing on PyPI • testPyPI  •  Upload Preparation •  Upload to testPyPI • Checking Project Page •  Testing our Package  •  Uploading on PyPI  •  Automating Build  • Going further

Product Documentation (Optional Topic)

How to document your product and generate the documentation with Sphinx
Documenting API • Sphinx Overview • Sphinx Quickstart • Sphinx Configuration • Documentation Layout • reST markup  • Generating Documentation  • Writing Good Documentation

Security Considerations (Optional Topic)

Some security considerations related to our Python product
Shopping on PyPI – The Challenges • Typosquatting Attacks • Looking for Candidates • Project Check List • Package Evaluation • Package Vulnerabilities Report • Securing Python Application • Python Security Cheat Sheet • Python Security Best Practice • Code Obfuscation