This commit is contained in:
Waylon Walker 2022-03-31 20:20:07 -05:00
commit 38355d2442
No known key found for this signature in database
GPG key ID: 66E2BF2B4190EFE4
9083 changed files with 1225834 additions and 0 deletions

View file

@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

View file

@ -0,0 +1,27 @@
.. _bool-ops:
Native boolean operations
=========================
Operations on ``bool`` values that are listed here have fast,
optimized implementations.
Construction
------------
* ``True``
* ``False``
* ``bool(obj)``
Operators
---------
* ``b1 and b2``
* ``b1 or b2``
* ``not b``
Functions
---------
* ``any(expr for ... in ...)``
* ``all(expr for ... in ...)``

View file

@ -0,0 +1,20 @@
.. _compilation-units:
Compilation units
=================
When you run mypyc to compile a set of modules, these modules form a
*compilation unit*. Mypyc will use early binding for references within
the compilation unit.
If you run mypyc multiple times to compile multiple sets of modules,
each invocation will result in a new compilation unit. References
between separate compilation units will fall back to late binding,
i.e. looking up names using Python namespace dictionaries. Also, all
calls will use the slower Python calling convention, where arguments
and the return value will be boxed (and potentially unboxed again in
the called function).
For maximal performance, minimize interactions across compilation
units. The simplest way to achieve this is to compile your entire
program as a single compilation unit.

View file

@ -0,0 +1,64 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
import sys
import os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('../..'))
from mypy.version import __version__ as mypy_version
# -- Project information -----------------------------------------------------
project = 'mypyc'
copyright = '2020, mypyc team'
author = 'mypyc team'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = mypy_version.split('-')[0]
# The full version, including alpha/beta/rc tags.
release = mypy_version
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [ # type: ignore
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
try:
import sphinx_rtd_theme # type: ignore
except:
html_theme = 'default'
else:
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

View file

@ -0,0 +1,25 @@
# Timings of CPython Operations
Here are some *very rough* approximate timings of CPython interpreter
operations:
* `f(1)` (empty function body): 70-90ns
* `f(n=1)` (empty function body): 90-110ns
* `o.x`: 30-40ns
* `o.f(1)` (empty method body): 80-160ns
* `Cls(1)` (initialize attribute in `__init__`): 290-330ns
* `x + y` (integers): 20-35ns
* `a[i]` (list) : 20-40ns
* `[i]` (also dealloc): 35-55ns
* `a.append(i)` (list, average over 5 appends): 70ns
* `d[s]` (dict, shared str key): 20ns
* `d[s] = i` (dict, shared str key): 40ns
* `isinstance(x, A)`: 100ns
* `(x, y)`: 20-35ns
* `x, y = t` (tuple expand): 10ns
Note that these results are very imprecise due to many factors, but
these should give a rough idea of the relative costs of various
operations.
Details: CPython 3.6.2, Macbook Pro 15" (Mid 2015), macOS Sierra

View file

@ -0,0 +1,548 @@
# Introduction for Mypyc Contributors
This is a short introduction aimed at anybody who is interested in
contributing to mypyc, or anybody who is curious to understand how
mypyc works internally.
## Key Differences from Python
Code compiled using mypyc is often much faster than CPython since it
does these things differently:
* Mypyc generates C that is compiled to native code, instead of
compiling to interpreted byte code, which CPython uses. Interpreted
byte code always has some interpreter overhead, which slows things
down.
* Mypyc doesn't let you arbitrarily monkey patch classes and functions
in compiled modules. This allows *early binding* -- mypyc
statically binds calls to compiled functions, instead of going
through a namespace dictionary. Mypyc can also call methods of
compiled classes using vtables, which are more efficient than
dictionary lookups used by CPython.
* Mypyc compiles classes to C extension classes, which are generally
more efficient than normal Python classes. They use an efficient,
fixed memory representation (essentially a C struct). This lets us
use direct memory access instead of (typically) two hash table
lookups to access an attribute.
* As a result of early binding, compiled code can use C calls to call
compiled functions. Keyword arguments can be translated to
positional arguments during compilation. Thus most calls to native
functions and methods directly map to simple C calls. CPython calls
are quite expensive, since mapping of keyword arguments, `*args`,
and so on has to mostly happen at runtime.
* Compiled code has runtime type checks to ensure that runtimes types
match the declared static types. Compiled code can thus make
assumptions about the types of expressions, resulting in both faster
and smaller code, since many runtime type checks performed by the
CPython interpreter can be omitted.
* Compiled code can often use unboxed (not heap allocated)
representations for integers, booleans and tuples.
## Supported Python Features
Mypyc supports a large subset of Python. Note that if you try to
compile something that is not supported, you may not always get a very
good error message.
Here are some major things that aren't yet supported in compiled code:
* Many dunder methods (only some work, such as `__init__` and `__eq__`)
* Monkey patching compiled functions or classes
* General multiple inheritance (a limited form is supported)
* Named tuple defined using the class-based syntax
* Defining protocols
We are generally happy to accept contributions that implement new Python
features.
## Development Environment
First you should set up the mypy development environment as described in
the [mypy docs](https://github.com/python/mypy/blob/master/README.md).
macOS, Linux and Windows are supported.
## Compiling and Running Programs
When working on a mypyc feature or a fix, you'll often need to run
compiled code. For example, you may want to do interactive testing or
to run benchmarks. This is also handy if you want to inspect the
generated C code (see Inspecting Generated C).
Run `mypyc` to compile a module to a C extension using your
development version of mypyc:
```
$ mypyc program.py
```
This will generate a C extension for `program` in the current working
directory. For example, on a Linux system the generated file may be
called `program.cpython-37m-x86_64-linux-gnu.so`.
Since C extensions can't be run as programs, use `python3 -c` to run
the compiled module as a program:
```
$ python3 -c "import program"
```
Note that `__name__` in `program.py` will now be `program`, not
`__main__`!
You can manually delete the C extension to get back to an interpreted
version (this example works on Linux):
```
$ rm program.*.so
```
Another option is to invoke mypyc through tests (see Testing below).
## High-level Overview of Mypyc
Mypyc compiles a Python module (or a set of modules) to C, and
compiles the generated C to a Python C extension module (or
modules). You can compile only a subset of your program to C --
compiled and interpreted code can freely and transparently
interact. You can also freely use any Python libraries (including C
extensions) in compiled code.
Mypyc will only make compiled code faster. To see a significant
speedup, you must make sure that most of the time is spent in compiled
code -- and not in libraries, for example.
Mypyc has these passes:
* Type check the code using mypy and infer types for variables and
expressions. This produces a mypy AST (defined in `mypy.nodes`) and
a type map that describes the inferred types (`mypy.types.Type`) of
all expressions (as PEP 484 types).
* Translate the mypy AST into a mypyc-specific intermediate representation (IR).
* The IR is defined in `mypyc.ir` (see below for an explanation of the IR).
* Various primitive operations used in the IR are defined in `mypyc.primitives`.
* The translation to IR happens in `mypyc.irbuild`. The top-level logic is in
`mypyc.irbuild.main`.
* Insert checks for uses of potentially uninitialized variables
(`mypyc.transform.uninit`).
* Insert exception handling (`mypyc.transform.exceptions`).
* Insert explicit reference count inc/dec opcodes (`mypyc.transform.refcount`).
* Translate the IR into C (`mypyc.codegen`).
* Compile the generated C code using a C compiler (`mypyc.build`).
## Useful Background Information
Beyond the mypy documentation, here are some things that are helpful to
know for mypyc contributors:
* Experience with C
([The C Programming Language](https://en.wikipedia.org/wiki/The_C_Programming_Language)
is a classic book about C)
* Basic familiarity with the Python C API (see
[Python C API documentation](https://docs.python.org/3/c-api/intro.html)). [Extending and Embedding the Python Interpreter](https://docs.python.org/3/extending/index.html) is a good tutorial for beginners.
* Basics of compilers (see the
[mypy wiki](https://github.com/python/mypy/wiki/Learning-Resources)
for some ideas)
## Mypyc Intermediate Representation (IR)
The mypyc IR is defined in `mypyc.ir`. It covers several key concepts
that are essential to understand by all mypyc contributors:
* `mypyc.ir.ops.Op` is an Abstract Base Class for all IR
operations. These are low-level and generally map to simple
fragments of C each. Mypy expressions are translated to
linear sequences of these ops.
* `mypyc.ir.ops.BasicBlock` is a container of a sequence of ops with a
branch/goto/return at the end, and no branch/goto/return ops in the
middle. Each function is compiled to a bunch of basic blocks.
* `mypyc.ir.rtypes.RType` and its subclasses are the types used for
everything in the IR. These are lower-level and simpler than mypy or
PEP 484 types. For example, there are no general-purpose generic
types types here. Each `List[X]` type (for any `X`) is represented
by a single `list` type, for example.
* Primitive types are special RTypes of which mypyc has some special
understanding, and there are typically some specialized
ops. Examples include `int` (referred to as `int_rprimitive` in the
code) and `list` (`list_rprimitive`). Python types for which there
is no specific RType type will be represented by the catch-all
`object_rprimitive` type.
* Instances of compiled classes are generally represented using the
`RInstance` type. Classes are compiled to C extension classes and
contain vtables for fast method calls and fast attribute access.
* IR representations of functions and classes live in
`mypyc.ir.func_ir` and `mypyc.ir.class_ir`, respectively.
Look at the docstrings and comments in `mypyc.ir` for additional
information. See the test cases in
`mypyc/test-data/irbuild-basic.test` for examples of what the IR looks
like in a pretty-printed form.
## Testing overview
Most mypyc test cases are defined in the same format (`.test`) as used
for test cases for mypy. Look at mypy developer documentation for a
general overview of how things work. Test cases live under
`mypyc/test-data/`, and you can run all mypyc tests via `pytest
-q mypyc`. If you don't make changes to code under `mypy/`, it's not
important to regularly run mypy tests during development.
When you create a PR, we have Continuous Integration jobs set up that
compile mypy using mypyc and run the mypy test suite using the
compiled mypy. This will sometimes catch additional issues not caught
by the mypyc test suite. It's okay to not do this in your local
development environment.
We discuss writing tests in more detail later in this document.
## Inspecting Generated IR
It's often useful to look at the generated IR when debugging issues or
when trying to understand how mypyc compiles some code. When you
compile some module by running `mypyc`, mypyc will write the
pretty-printed IR into `build/ops.txt`. This is the final IR that
includes the output from exception and reference count handling
insertion passes.
We also have tests that verify the generate IR
(`mypyc/test-data/irbuild-*.text`).
## Type-checking Mypyc
`./runtests.py self` type checks mypy and mypyc. This is pretty slow,
however, since it's using an uncompiled mypy.
Installing a released version of mypy using `pip` (which is compiled)
and using `dmypy` (mypy daemon) is a much, much faster way to type
check mypyc during development.
## Value Representation
Mypyc uses a tagged pointer representation for values of type `int`
(`CPyTagged`), `char` for booleans, and C structs for tuples. For most
other objects mypyc uses the CPython `PyObject *`.
Python integers that fit in 31/63 bits (depending on whether we are on
a 32-bit or 64-bit platform) are represented as C integers
(`CPyTagged`) shifted left by 1. Integers that don't fit in this
representation are represented as pointers to a `PyObject *` (this is
always a Python `int` object) with the least significant bit
set. Tagged integer operations are defined in `mypyc/lib-rt/int_ops.c`
and `mypyc/lib-rt/CPy.h`.
There are also low-level integer types, such as `int32` (see
`mypyc.ir.rtypes`), that don't use the tagged representation. These
types are not exposed to users, but they are used in generated code.
## Overview of Generated C
Mypyc compiles a function into two functions, a native function and
a wrapper function:
* The native function takes a fixed number of C arguments with the
correct C types. It assumes that all argument have correct types.
* The wrapper function conforms to the Python C API calling convention
and takes an arbitrary set of arguments. It processes the arguments,
checks their types, unboxes values with special representations and
calls the native function. The return value from the native function
is translated back to a Python object ("boxing").
Calls to other compiled functions don't go through the Python module
namespace but directly call the target native C function. This makes
calls very fast compared to CPython.
The generated code does runtime checking so that it can assume that
values always have the declared types. Whenever accessing CPython
values which might have unexpected types we need to insert a runtime
type check operation. For example, when getting a list item we need to
insert a runtime type check (an unbox or a cast operation), since
Python lists can contain arbitrary objects.
The generated code uses various helpers defined in
`mypyc/lib-rt/CPy.h`. The implementations are in various `.c` files
under `mypyc/lib-rt`.
## Inspecting Generated C
It's often useful to inspect the C code genenerate by mypyc to debug
issues. Mypyc stores the generated C code as `build/__native.c`.
Compiled native functions have the prefix `CPyDef_`, while wrapper
functions used for calling functions from interpreted Python code have
the `CPyPy_` prefix.
## Other Important Limitations
All of these limitations will likely be fixed in the future:
* We don't detect stack overflows.
* We don't handle Ctrl-C in compiled code.
## Hints for Implementing Typical Mypyc Features
This section gives an overview of where to look for and
what to do to implement specific kinds of mypyc features.
### Testing
Our bread-and-butter testing strategy is compiling code with mypyc and
running it. There are downsides to this (kind of slow, tests a huge
number of components at once, insensitive to the particular details of
the IR), but there really is no substitute for running code. You can
also write tests that test the generated IR, however.
### Tests that compile and run code
Test cases that compile and run code are located in
`mypyc/test-data/run*.test` and the test runner is in
`mypyc.test.test_run`. The code to compile comes after `[case
test<name>]`. The code gets saved into the file `native.py`, and it
gets compiled into the module `native`.
Each test case uses a non-compiled Python driver that imports the
`native` module and typically calls some compiled functions. Some
tests also perform assertions and print messages in the driver.
If you don't provide a driver, a default driver is used. The default
driver just calls each module-level function that is prefixed with
`test_` and reports any uncaught exceptions as failures. (Failure to
build or a segfault also count as failures.) `testStringOps` in
`mypyc/test-data/run-strings.test` is an example of a test that uses
the default driver.
You should usually use the default driver (don't include
`driver.py`). It's the simplest way to write most tests.
Here's an example test case that uses the default driver:
```
[case testConcatenateLists]
def test_concat_lists() -> None:
assert [1, 2] + [5, 6] == [1, 2, 5, 6]
def test_concat_empty_lists() -> None:
assert [] + [] == []
```
There is one test case, `testConcatenateLists`. It has two sub-cases,
`test_concat_lists` and `test_concat_empty_lists`. Note that you can
use the pytest -k argument to only run `testConcetanateLists`, but you
can't filter tests at the sub-case level.
It's recommended to have multiple sub-cases per test case, since each
test case has significant fixed overhead. Each test case is run in a
fresh Python subprocess.
Many of the existing test cases provide a custom driver by having
`[file driver.py]`, followed by the driver implementation. Here the
driver is not compiled, which is useful if you want to test
interactions between compiled and non-compiled code. However, many of
the tests don't have a good reason to use a custom driver -- when they
were written, the default driver wasn't available.
Test cases can also have a `[out]` section, which specifies the
expected contents of stdout the test case should produce. New test
cases should prefer assert statements to `[out]` sections.
### IR tests
If the specifics of the generated IR of a change is important
(because, for example, you want to make sure a particular optimization
is triggering), you should add a `mypyc.irbuild` test as well. Test
cases are located in `mypyc/test-data/irbuild-*.test` and the test
driver is in `mypyc.test.test_irbuild`. IR build tests do a direct
comparison of the IR output, so try to make the test as targeted as
possible so as to capture only the important details. (Many of our
existing IR build tests do not follow this advice, unfortunately!)
If you pass the `--update-data` flag to pytest, it will automatically
update the expected output of any tests to match the actual
output. This is very useful for changing or creating IR build tests,
but make sure to carefully inspect the diff!
You may also need to add some definitions to the stubs used for
builtins during tests (`mypyc/test-data/fixtures/ir.py`). We don't use
full typeshed stubs to run tests since they would seriously slow down
tests.
### Benchmarking
Many mypyc improvements attempt to make some operations faster. For
any such change, you should run some measurements to verify that
there actually is a measurable performance impact.
A typical benchmark would initialize some data to be operated on, and
then measure time spent in some function. In particular, you should
not measure time needed to run the entire benchmark program, as this
would include Python startup overhead and other things that aren't
relevant. In general, for microbenchmarks, you want to do as little as
possible in the timed portion. So ideally you'll just have some loops
and the code under test. Be ready to provide your benchmark in code
review so that mypyc developers can check that the benchmark is fine
(writing a good benchmark is non-trivial).
You should run a benchmark at least five times, in both original and
changed versions, ignore outliers, and report the average
runtime. Actual performance of a typical desktop or laptop computer is
quite variable, due to dynamic CPU clock frequency changes, background
processes, etc. If you observe a high variance in timings, you'll need
to run the benchmark more times. Also try closing most applications,
including web browsers.
Interleave original and changed runs. Don't run 10 runs with variant A
followed by 10 runs with variant B, but run an A run, a B run, an A
run, etc. Otherwise you risk that the CPU frequency will be different
between variants. You can also try adding a delay of 5 to 20s between
runs to avoid CPU frequency changes.
Instead of averaging over many measurements, you can try to adjust
your environment to provide more stable measurements. However, this
can be hard to do with some hardware, including many laptops. Victor
Stinner has written a series of blog posts about making measurements
stable:
* https://vstinner.github.io/journey-to-stable-benchmark-system.html
* https://vstinner.github.io/journey-to-stable-benchmark-average.html
### Adding C Helpers
If you add an operation that compiles into a lot of C code, you may
also want to add a C helper function for the operation to make the
generated code smaller. Here is how to do this:
* Declare the operation in `mypyc/lib-rt/CPy.h`. We avoid macros, and
we generally avoid inline functions to make it easier to target
additional backends in the future.
* Consider adding a unit test for your C helper in `mypyc/lib-rt/test_capi.cc`.
We use
[Google Test](https://github.com/google/googletest) for writing
tests in C++. The framework is included in the repository under the
directory `googletest/`. The C unit tests are run as part of the
pytest test suite (`test_c_unit_test`).
### Adding a Specialized Primitive Operation
Mypyc speeds up operations on primitive types such as `list` and `int`
by having primitive operations specialized for specific types. These
operations are declared in `mypyc.primitives` (and
`mypyc/lib-rt/CPy.h`). For example, `mypyc.primitives.list_ops`
contains primitives that target list objects.
The operation definitions are data driven: you specify the kind of
operation (such as a call to `builtins.len` or a binary addition) and
the operand types (such as `list_primitive`), and what code should be
generated for the operation. Mypyc does AST matching to find the most
suitable primitive operation automatically.
Look at the existing primitive definitions and the docstrings in
`mypyc.primitives.registry` for examples and more information.
### Adding a New Primitive Type
Some types (typically Python Python built-in types), such as `int` and
`list`, are special cased in mypyc to generate optimized operations
specific to these types. We'll occasionally want to add additional
primitive types.
Here are some hints about how to add support for a new primitive type
(this may be incomplete):
* Decide whether the primitive type has an "unboxed" representation (a
representation that is not just `PyObject *`). For most types we'll
use a boxed representation, as it's easier to implement and more
closely matches Python semantics.
* Create a new instance of `RPrimitive` to support the primitive type
and add it to `mypyc.ir.rtypes`. Make sure all the attributes are
set correctly and also define `<foo>_rprimitive` and
`is_<foo>_rprimitive`.
* Update `mypyc.irbuild.mapper.Mapper.type_to_rtype()`.
* If the type is not unboxed, update `emit_cast` in `mypyc.codegen.emit`.
If the type is unboxed, there are some additional steps:
* Update `emit_box` in `mypyc.codegen.emit`.
* Update `emit_unbox` in `mypyc.codegen.emit`.
* Update `emit_inc_ref` and `emit_dec_ref` in `mypypc.codegen.emit`.
If the unboxed representation does not need reference counting,
these can be no-ops.
* Update `emit_error_check` in `mypyc.codegen.emit`.
* Update `emit_gc_visit` and `emit_gc_clear` in `mypyc.codegen.emit`
if the type has an unboxed representation with pointers.
The above may be enough to allow you to declare variables with the
type, pass values around, perform runtime type checks, and use generic
fallback primitive operations to perform method calls, binary
operations, and so on. You likely also want to add some faster,
specialized primitive operations for the type (see Adding a
Specialized Primitive Operation above for how to do this).
Add a test case to `mypyc/test-data/run*.test` to test compilation and
running compiled code. Ideas for things to test:
* Test using the type as an argument.
* Test using the type as a return value.
* Test passing a value of the type to a function both within
compiled code and from regular Python code. Also test this
for return values.
* Test using the type as list item type. Test both getting a list item
and setting a list item.
### Supporting More Python Syntax
Mypyc supports most Python syntax, but there are still some gaps.
Support for syntactic sugar that doesn't need additional IR operations
typically only requires changes to `mypyc.irbuild`.
Some new syntax also needs new IR primitives to be added to
`mypyc.primitives`. See `mypyc.primitives.registry` for documentation
about how to do this.
### Other Hints
* This developer documentation is not aimed to be very complete. Much
of our documentation is in comments and docstring in the code. If
something is unclear, study the code.
* It can be useful to look through some recent PRs to get an idea of
what typical code changes, test cases, etc. look like.
* Feel free to open GitHub issues with questions if you need help when
contributing, or ask questions in existing issues. Note that we only
support contributors. Mypyc is not (yet) an end-user product. You
can also ask questions in our Gitter chat
(https://gitter.im/mypyc-dev/community).
## Undocumented Workflows
These workflows would be useful for mypyc contributors. We should add
them to mypyc developer documentation:
* How to inspect the generated IR before some transform passes.

View file

@ -0,0 +1,59 @@
.. _dict-ops:
Native dict operations
======================
These ``dict`` operations have fast, optimized implementations. Other
dictionary operations use generic implementations that are often slower.
Construction
------------
Construct dict from keys and values:
* ``{key: value, ...}``
Construct empty dict:
* ``{}``
* ``dict()``
Construct dict from another object:
* ``dict(d: dict)``
* ``dict(x: Iterable)``
Dict comprehensions:
* ``{...: ... for ... in ...}``
* ``{...: ... for ... in ... if ...}``
Operators
---------
* ``d[key]``
* ``value in d``
Statements
----------
* ``d[key] = value``
* ``for key in d:``
Methods
-------
* ``d.get(key)``
* ``d.get(key, default)``
* ``d.keys()``
* ``d.values()``
* ``d.items()``
* ``d.copy()``
* ``d.clear()``
* ``d1.update(d2: dict)``
* ``d.update(x: Iterable)``
Functions
---------
* ``len(d: dict)``

View file

@ -0,0 +1,281 @@
.. _differences-from-python:
Differences from Python
=======================
Mypyc aims to be sufficiently compatible with Python semantics so that
migrating code to mypyc often doesn't require major code
changes. There are various differences to enable performance gains
that you need to be aware of, however.
This section documents notable differences from Python. We discuss
many of them also elsewhere, but it's convenient to have them here in
one place.
Running compiled modules
------------------------
You can't use ``python3 <module>.py`` or ``python3 -m <module>``
to run compiled modules. Use ``python3 -c "import <module>"`` instead,
or write a wrapper script that imports your module.
As a side effect, you can't rely on checking the ``__name__`` attribute in compiled
code, like this::
if __name__ == "__main__": # Can't be used in compiled code
main()
Type errors prevent compilation
-------------------------------
You can't compile code that generates mypy type check errors. You can
sometimes ignore these with a ``# type: ignore`` comment, but this can
result in bad code being generated, and it's considered dangerous.
.. note::
In the future, mypyc may reject ``# type: ignore`` comments that
may be unsafe.
Runtime type checking
---------------------
Non-erased types in annotations will be type checked at runtime. For example,
consider this function::
def twice(x: int) -> int:
return x * 2
If you try to call this function with a ``float`` or ``str`` argument,
you'll get a type error on the call site, even if the call site is not
being type checked::
twice(5) # OK
twice(2.2) # TypeError
twice("blah") # TypeError
Also, values with *inferred* types will be type checked. For example,
consider a call to the stdlib function ``socket.gethostname()`` in
compiled code. This function is not compiled (no stdlib modules are
compiled with mypyc), but mypyc uses a *library stub file* to infer
the return type as ``str``. Compiled code calling ``gethostname()``
will fail with ``TypeError`` if ``gethostname()`` would return an
incompatible value, such as ``None``::
import socket
# Fail if returned value is not a str
name = socket.gethostname()
Note that ``gethostname()`` is defined like this in the stub file for
``socket`` (in typeshed)::
def gethostname() -> str: ...
Thus mypyc verifies that library stub files and annotations in
non-compiled code match runtime values. This adds an extra layer of
type safety.
Casts such as ``cast(str, x)`` will also result in strict type
checks. Consider this example::
from typing import cast
...
x = cast(str, y)
The last line is essentially equivalent to this Python code when compiled::
if not isinstance(y, str):
raise TypeError(...)
x = y
In interpreted mode ``cast`` does not perform a runtime type check.
Native classes
--------------
Native classes behave differently from Python classes. See
:ref:`native-classes` for the details.
Primitive types
---------------
Some primitive types behave differently in compiled code to improve
performance.
``int`` objects use an unboxed (non-heap-allocated) representation for small
integer values. A side effect of this is that the exact runtime type of
``int`` values is lost. For example, consider this simple function::
def first_int(x: List[int]) -> int:
return x[0]
print(first_int([True])) # Output is 1, instead of True!
``bool`` is a subclass of ``int``, so the above code is
valid. However, when the list value is converted to ``int``, ``True``
is converted to the corresponding ``int`` value, which is ``1``.
Note that integers still have an arbitrary precision in compiled code,
similar to normal Python integers.
Fixed-length tuples are unboxed, similar to integers. The exact type
and identity of fixed-length tuples is not preserved, and you can't
reliably use ``is`` checks to compare tuples that are used in compiled
code.
.. _early-binding:
Early binding
-------------
References to functions, types, most attributes, and methods in the
same :ref:`compilation unit <compilation-units>` use *early binding*:
the target of the reference is decided at compile time, whenever
possible. This contrasts with normal Python behavior of *late
binding*, where the target is found by a namespace lookup at
runtime. Omitting these namespace lookups improves performance, but
some Python idioms don't work without changes.
Note that non-final module-level variables still use late binding.
You may want to avoid these in very performance-critical code.
Examples of early and late binding::
from typing import Final
import lib # "lib" is not compiled
x = 0
y: Final = 1
def func() -> None:
pass
class Cls:
def __init__(self, attr: int) -> None:
self.attr = attr
def method(self) -> None:
pass
def example() -> None:
# Early binding:
var = y
func()
o = Cls()
o.x
o.method()
# Late binding:
var = x # Module-level variable
lib.func() # Accessing library that is not compiled
Monkey patching
---------------
Since mypyc function and class definitions are immutable, you can't
perform arbitrary monkey patching, such as replacing functions or
methods with mocks in tests.
.. note::
Each compiled module has a Python namespace that is initialized to
point to compiled functions and type objects. This namespace is a
regular ``dict`` object, and it *can* be modified. However,
compiled code generally doesn't use this namespace, so any changes
will only be visible to non-compiled code.
Stack overflows
---------------
Compiled code currently doesn't check for stack overflows. Your
program may crash in an unrecoverable fashion if you have too many
nested function calls, typically due to out-of-control recursion.
.. note::
This limitation will be fixed in the future.
Final values
------------
Compiled code replaces a reference to an attribute declared ``Final`` with
the value of the attribute computed at compile time. This is an example of
:ref:`early binding <early-binding>`. Example::
MAX: Final = 100
def limit_to_max(x: int) -> int:
if x > MAX:
return MAX
return x
The two references to ``MAX`` don't involve any module namespace lookups,
and are equivalent to this code::
def limit_to_max(x: int) -> int:
if x > 100:
return 100
return x
When run as interpreted, the first example will execute slower due to
the extra namespace lookups. In interpreted code final attributes can
also be modified.
Unsupported features
--------------------
Some Python features are not supported by mypyc (yet). They can't be
used in compiled code, or there are some limitations. You can
partially work around some of these limitations by running your code
in interpreted mode.
Operator overloading
********************
Native classes can only use these dunder methods to override operators:
* ``__eq__``
* ``__ne__``
* ``__getitem__``
* ``__setitem__``
.. note::
This limitation will be lifted in the future.
Generator expressions
*********************
Generator expressions are not supported. To make it easier to compile
existing code, they are implicitly replaced with list comprehensions.
*This does not always produce the same behavior.*
To work around this limitation, you can usually use a generator
function instead. You can sometimes replace the generator expression
with an explicit list comprehension.
Descriptors
***********
Native classes can't contain arbitrary descriptors. Properties, static
methods and class methods are supported.
Stack introspection
*******************
Frames of compiled functions can't be inspected using ``inspect``.
Profiling hooks and tracing
***************************
Compiled functions don't trigger profiling and tracing hooks, such as
when using the ``profile``, ``cProfile``, or ``trace`` modules.
Debuggers
*********
You can't set breakpoints in compiled functions or step through
compiled functions using ``pdb``. Often you can debug your code in
interpreted mode instead.

View file

@ -0,0 +1,24 @@
.. _float-ops:
Native float operations
========================
These ``float`` operations have fast, optimized implementations. Other
floating point operations use generic implementations that are often
slower.
.. note::
At the moment, only a few float operations are optimized. This will
improve in future mypyc releases.
Construction
------------
* Float literal
* ``float(string)``
Functions
---------
* ``abs(f)``

View file

@ -0,0 +1,42 @@
# Future
This document introduces some ideas for future improvements.
## Basic Optimizations
Implement basic optimizations such as common subexpression elimination and
loop invariant code motion.
Importantly, common subexpression elimination could be used to avoid
redundant type checks.
## Operation-specific Optimizations
Some operations or combinations of successive operations can be
replaced with more efficient operations. Examples:
* If `s` is a string, `s[i] == 'x'` doesn't need to construct the
intermediate single-character string object `s[i]` but just compare
the character value to `ord('x')`.
* `a + ':' + b` (two string concetenations) can be implemented as
single three-operand concatenation that doesn't construct an
intermediate object.
* `x in {1, 3}` can be translated into `x == 1 or x == 3` (more
generally we need to evaluate all right-hand-side items).
## Integer Range Analysis
Implement integer range analysis. This can be used in various ways:
* Use untagged representations for some registers.
* Use faster integer arithmetic operations for operations that
only deal with short integers or that can't overflow.
* Remove redundant list and string index checks.
## Always Defined Attributes
Somehow make it possible to enforce that attributes in a class are always
defined. This makes attribute access faster since we don't need to explicitly
check if the attribute is defined.

View file

@ -0,0 +1,240 @@
Getting started
===============
Here you will learn some basic things you need to know to get started with mypyc.
Prerequisites
-------------
You need a Python C extension development environment. The way to set this up
depends on your operating system.
macOS
*****
Install Xcode command line tools:
.. code-block::
$ xcode-select --install
Linux
*****
You need a C compiler and CPython headers and libraries. The specifics
of how to install these varies by distribution. Here are instructions for
Ubuntu 18.04, for example:
.. code-block::
$ sudo apt install python3-dev
Windows
*******
Install `Visual C++ <https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017>`_.
Installation
------------
Mypyc is shipped as part of the mypy distribution. Install mypy like
this (you need Python 3.5 or later):
.. code-block::
$ python3 -m pip install -U mypy
On some systems you need to use this instead:
.. code-block::
$ python -m pip install -U mypy
Example program
---------------
Let's start with a classic micro-benchmark, recursive fibonacci. Save
this file as ``fib.py``:
.. code-block:: python
import time
def fib(n: int) -> int:
if n <= 1:
return n
else:
return fib(n - 2) + fib(n - 1)
t0 = time.time()
fib(32)
print(time.time() - t0)
Note that we gave the ``fib`` function a type annotation. Without it,
performance won't be as impressive after compilation.
.. note::
`Mypy documentation
<https://mypy.readthedocs.io/en/stable/index.html>`_ is a good
introduction if you are new to type annotations or mypy. Mypyc uses
mypy to perform type checking and type inference, so some familiarity
with mypy is very useful.
Compiling and running
---------------------
We can run ``fib.py`` as a regular, interpreted program using CPython:
.. code-block:: console
$ python3 fib.py
0.4125328063964844
It took about 0.41s to run on my computer.
Run ``mypyc`` to compile the program to a binary C extension:
.. code-block:: console
$ mypyc fib.py
This will generate a C extension for ``fib`` in the current working
directory. For example, on a Linux system the generated file may be
called ``fib.cpython-37m-x86_64-linux-gnu.so``.
Since C extensions can't be run as programs, use ``python3 -c`` to run
the compiled module as a program:
.. code-block:: console
$ python3 -c "import fib"
0.04097270965576172
After compilation, the program is about 10x faster. Nice!
.. note::
``__name__`` in ``fib.py`` would now be ``"fib"``, not ``"__main__"``.
You can also pass most
`mypy command line options <https://mypy.readthedocs.io/en/stable/command_line.html>`_
to ``mypyc``.
Deleting compiled binary
------------------------
You can manually delete the C extension to get back to an interpreted
version (this example works on Linux):
.. code-block::
$ rm fib.*.so
Using setup.py
--------------
You can also use ``setup.py`` to compile modules using mypyc. Here is an
example ``setup.py`` file::
from setuptools import setup
from mypyc.build import mypycify
setup(
name='mylib',
packages=['mylib'],
ext_modules=mypycify([
'mylib/__init__.py',
'mylib/mod.py',
]),
)
We used ``mypycify(...)`` to specify which files to compile using
mypyc. Your ``setup.py`` can include additional Python files outside
``mypycify(...)`` that won't be compiled.
Now you can build a wheel (.whl) file for the package::
python3 setup.py bdist_wheel
The wheel is created under ``dist/``.
You can also compile the C extensions in-place, in the current directory (similar
to using ``mypyc`` to compile modules)::
python3 setup.py build_ext --inplace
You can include most `mypy command line options
<https://mypy.readthedocs.io/en/stable/command_line.html>`_ in the
list of arguments passed to ``mypycify()``. For example, here we use
the ``--disallow-untyped-defs`` flag to require that all functions
have type annotations::
...
setup(
name='frobnicate',
packages=['frobnicate'],
ext_modules=mypycify([
'--disallow-untyped-defs', # Pass a mypy flag
'frobnicate.py',
]),
)
.. note:
You may be tempted to use `--check-untyped-defs
<https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-check-untyped-defs>`_
to type check functions without type annotations. Note that this
may reduce performance, due to many transitions between type-checked and unchecked
code.
Recommended workflow
--------------------
A simple way to use mypyc is to always compile your code after any
code changes, but this can get tedious, especially if you have a lot
of code. Instead, you can do most development in interpreted mode.
This development workflow has worked smoothly for developing mypy and
mypyc (often we forget that we aren't working on a vanilla Python
project):
1. During development, use interpreted mode. This gives you a fast
edit-run cycle.
2. Use type annotations liberally and use mypy to type check your code
during development. Mypy and tests can find most errors that would
break your compiled code, if you have good type annotation
coverage. (Running mypy is pretty quick.)
3. After you've implemented a feature or a fix, compile your project
and run tests again, now in compiled mode. Usually nothing will
break here, assuming your type annotation coverage is good. This
can happen locally or in a Continuous Integration (CI) job. If you
have CI, compiling locally may be rarely needed.
4. Release or deploy a compiled version. Optionally, include a
fallback interpreted version for platforms that mypyc doesn't
support.
This mypyc workflow only involves minor tweaks to a typical Python
workflow. Most of development, testing and debugging happens in
interpreted mode. Incremental mypy runs, especially when using the
mypy daemon, are very quick (often a few hundred milliseconds).
Next steps
----------
You can sometimes get good results by just annotating your code and
compiling it. If this isn't providing meaningful performance gains, if
you have trouble getting your code to work under mypyc, or if you want
to optimize your code for maximum performance, you should read the
rest of the documentation in some detail.
Here are some specific recommendations, or you can just read the
documentation in order:
* :ref:`using-type-annotations`
* :ref:`native-classes`
* :ref:`differences-from-python`
* :ref:`performance-tips`

View file

@ -0,0 +1,55 @@
.. mypyc documentation master file, created by
sphinx-quickstart on Sun Apr 5 14:01:55 2020.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to mypyc documentation!
===============================
Mypyc compiles Python modules to C extensions. It uses standard Python
`type hints
<https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html>`_ to
generate fast code.
.. toctree::
:maxdepth: 2
:caption: First steps
introduction
getting_started
.. toctree::
:maxdepth: 2
:caption: Using mypyc
using_type_annotations
native_classes
differences_from_python
compilation_units
.. toctree::
:maxdepth: 2
:caption: Native operations reference
native_operations
int_operations
bool_operations
float_operations
str_operations
list_operations
dict_operations
set_operations
tuple_operations
.. toctree::
:maxdepth: 2
:caption: Advanced topics
performance_tips_and_tricks
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

View file

@ -0,0 +1,34 @@
.. _int-ops:
Native integer operations
=========================
Operations on ``int`` values that are listed here have fast, optimized
implementations. Other integer operations use generic implementations
that are often slower. Some operations involving integers and other
types are documented elsewhere, such as list indexing.
Construction
------------
* Integer literal
* ``int(x: float)``
* ``int(x: str)``
* ``int(x: str, base: int)``
Operators
---------
* Arithmetic (``+``, ``-``, ``*``, ``//``, ``%``)
* Bitwise operations (``&``, ``|``, ``^``, ``<<``, ``>>``, ``~``)
* Comparisons (``==``, ``!=``, ``<``, etc.)
* Augmented assignment (``x += y``, etc.)
Statements
----------
For loop over range:
* ``for x in range(end)``
* ``for x in range(start, end)``
* ``for x in range(start, end, step)``

View file

@ -0,0 +1,150 @@
Introduction
============
Mypyc compiles Python modules to C extensions. It uses standard Python
`type hints
<https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html>`_ to
generate fast code.
The compiled language is a strict, *gradually typed* Python variant. It
restricts the use of some dynamic Python features to gain performance,
but it's mostly compatible with standard Python.
Mypyc uses `mypy <http://www.mypy-lang.org/>`_ to perform type
checking and type inference. Most type system features in the stdlib
`typing <https://docs.python.org/3/library/typing.html>`_ module are
supported.
Compiled modules can import arbitrary Python modules and third-party
libraries. You can compile anything from a single performance-critical
module to your entire codebase. You can run the modules you compile
also as normal, interpreted Python modules.
Existing code with type annotations is often **1.5x to 5x** faster
when compiled. Code tuned for mypyc can be **5x to 10x** faster.
Mypyc currently aims to speed up non-numeric code, such as server
applications. Mypyc is also used to compile itself (and mypy).
Why mypyc?
----------
**Easy to get started.** Compiled code has the look and feel of
regular Python code. Mypyc supports familiar Python syntax and idioms.
**Expressive types.** Mypyc fully supports standard Python type hints.
Mypyc has local type inference, generics, optional types, tuple types,
union types, and more. Type hints act as machine-checked
documentation, making code not only faster but also easier to
understand and modify.
**Python ecosystem.** Mypyc runs on top of CPython, the
standard Python implementation. You can use any third-party libraries,
including C extensions, installed with pip. Mypyc uses only valid Python
syntax, so all Python editors and IDEs work perfectly.
**Fast program startup.** Mypyc uses ahead-of-time compilation, so
compilation does not slow down program startup. Slow program startup
is a common issue with JIT compilers.
**Migration path for existing code.** Existing Python code often
requires only minor changes to compile using mypyc.
**Waiting for compilation is optional.** Compiled code also runs as
normal Python code. You can use interpreted Python during development,
with familiar and fast workflows.
**Runtime type safety.** Mypyc protects you from segfaults and memory
corruption. Any unexpected runtime type safety violation is a bug in
mypyc. Runtime values are checked against type annotations. (Without
mypyc, type annotations are ignored at runtime.)
**Find errors statically.** Mypyc uses mypy for static type checking
that helps catch many bugs.
Use cases
---------
**Fix only performance bottlenecks.** Often most time is spent in a few
Python modules or functions. Add type annotations and compile these
modules for easy performance gains.
**Compile it all.** During development you can use interpreted mode,
for a quick edit-run cycle. In releases all non-test code is compiled.
This is how mypy achieved a 4x performance improvement over interpreted
Python.
**Take advantage of existing type hints.** If you already use type
annotations in your code, adopting mypyc will be easier. You've already
done most of the work needed to use mypyc.
**Alternative to a lower-level language.** Instead of writing
performance-critical code in C, C++, Cython or Rust, you may get good
performance while staying in the comfort of Python.
**Migrate C extensions.** Maintaining C extensions is not always fun
for a Python developer. With mypyc you may get performance similar to
the original C, with the convenience of Python.
Differences from Cython
-----------------------
Mypyc targets many similar use cases as Cython. Mypyc does many things
differently, however:
* No need to use non-standard syntax, such as ``cpdef``, or extra
decorators to get good performance. Clean, normal-looking
type-annotated Python code can be fast without language extensions.
This makes it practical to compile entire codebases without a
developer productivity hit.
* Mypyc has first-class support for features in the ``typing`` module,
such as tuple types, union types and generics.
* Mypyc has powerful type inference, provided by mypy. Variable type
annotations are not needed for optimal performance.
* Mypyc fully integrates with mypy for robust and seamless static type
checking.
* Mypyc performs strict enforcement of type annotations at runtime,
resulting in better runtime type safety and easier debugging.
Unlike Cython, mypyc doesn't directly support interfacing with C libraries
or speeding up numeric code.
How does it work
----------------
Mypyc uses several techniques to produce fast code:
* Mypyc uses *ahead-of-time compilation* to native code. This removes
CPython interpreter overhead.
* Mypyc enforces type annotations (and type comments) at runtime,
raising ``TypeError`` if runtime values don't match annotations.
Value types only need to be checked in the boundaries between
dynamic and static typing.
* Compiled code uses optimized, type-specific primitives.
* Mypyc uses *early binding* to resolve called functions and name
references at compile time. Mypyc avoids many dynamic namespace
lookups.
* Classes are compiled to *C extension classes*. They use `vtables
<https://en.wikipedia.org/wiki/Virtual_method_table>`_ for fast
method calls and attribute access.
* Mypyc treats compiled functions, classes, and attributes declared
``Final`` as immutable.
* Mypyc has memory-efficient, unboxed representions for integers and
booleans.
Development status
------------------
Mypyc is currently alpha software. It's only recommended for
production use cases with careful testing, and if you are willing to
contribute fixes or to work around issues you will encounter.

View file

@ -0,0 +1,65 @@
.. _list-ops:
Native list operations
======================
These ``list`` operations have fast, optimized implementations. Other
list operations use generic implementations that are often slower.
Construction
------------
Construct list with specific items:
* ``[item0, ..., itemN]``
Construct empty list:
* ``[]``
* ``list()``
Construct list from iterable:
* ``list(x: Iterable)``
List comprehensions:
* ``[... for ... in ...]``
* ``[... for ... in ... if ...]``
Operators
---------
* ``lst[n]`` (get item by integer index)
* ``lst[n:m]``, ``lst[n:]``, ``lst[:m]``, ``lst[:]`` (slicing)
* ``lst * n``, ``n * lst``
* ``obj in lst``
Statements
----------
Set item by integer index:
* ``lst[n] = x``
For loop over a list:
* ``for item in lst:``
Methods
-------
* ``lst.append(obj)``
* ``lst.extend(x: Iterable)``
* ``lst.insert(index, obj)``
* ``lst.pop(index=-1)``
* ``lst.remove(obj)``
* ``lst.count(obj)``
* ``lst.index(obj)``
* ``lst.reverse()``
* ``lst.sort()``
Functions
---------
* ``len(lst: list)``

View file

@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd

View file

@ -0,0 +1,198 @@
.. _native-classes:
Native classes
==============
Classes in compiled modules are *native classes* by default (some
exceptions are discussed below). Native classes are compiled to C
extension classes, which have some important differences from normal
Python classes. Native classes are similar in many ways to built-in
types, such as ``int``, ``str``, and ``list``.
Immutable namespaces
--------------------
The type object namespace of native classes is mostly immutable (but
class variables can be assigned to)::
class Cls:
def method1(self) -> None:
print("method1")
def method2(self) -> None:
print("method2")
Cls.method1 = Cls.method2 # Error
Cls.new_method = Cls.method2 # Error
Only attributes defined within a class definition (or in a base class)
can be assigned to (similar to using ``__slots__``)::
class Cls:
x: int
def __init__(self, y: int) -> None:
self.x = 0
self.y = y
def method(self) -> None:
self.z = "x"
o = Cls(0)
print(o.x, o.y) # OK
o.z = "y" # OK
o.extra = 3 # Error: no attribute "extra"
.. _inheritance:
Inheritance
-----------
Only single inheritance is supported (except for :ref:`traits
<trait-types>`). Most non-native classes can't be used as base
classes.
These non-native classes can be used as base classes of native
classes:
* ``object``
* ``dict`` (and ``Dict[k, v]``)
* ``BaseException``
* ``Exception``
* ``ValueError``
* ``IndexError``
* ``LookupError``
* ``UserWarning``
By default, a non-native class can't inherit a native class, and you
can't inherit from a native class outside the compilation unit that
defines the class. You can enable these through
``mypy_extensions.mypyc_attr``::
from mypy_extensions import mypyc_attr
@mypyc_attr(allow_interpreted_subclasses=True)
class Cls:
...
Allowing interpreted subclasses has only minor impact on performance
of instances of the native class. Accessing methods and attributes of
a *non-native* subclass (or a subclass defined in another compilation
unit) will be slower, since it needs to use the normal Python
attribute access mechanism.
You need to install ``mypy-extensions`` to use ``@mypyc_attr``:
.. code-block:: text
pip install --upgrade mypy-extensions
Class variables
---------------
Class variables must be explicitly declared using ``attr: ClassVar``
or ``attr: ClassVar[<type>]``. You can't assign to a class variable
through an instance. Example::
from typing import ClassVar
class Cls:
cv: ClassVar = 0
Cls.cv = 2 # OK
o = Cls()
print(o.cv) # OK (2)
o.cv = 3 # Error!
Generic native classes
----------------------
Native classes can be generic. Type variables are *erased* at runtime,
and instances don't keep track of type variable values.
Compiled code thus can't check the values of type variables when
performing runtime type checks. These checks are delayed to when
reading a value with a type variable type::
from typing import TypeVar, Generic, cast
T = TypeVar('T')
class Box(Generic[T]):
def __init__(self, item: T) -> None:
self.item = item
x = Box(1) # Box[int]
y = cast(Box[str], x) # OK (type variable value not checked)
y.item # Runtime error: item is "int", but "str" expected
Metaclasses
-----------
Most metaclasses aren't supported with native classes, since their
behavior is too dynamic. You can use these metaclasses, however:
* ``abc.ABCMeta``
* ``typing.GenericMeta`` (used by ``typing.Generic``)
.. note::
If a class definition uses an unsupported metaclass, *mypyc
compiles the class into a regular Python class*.
Class decorators
----------------
Similar to metaclasses, most class decorators aren't supported with
native classes, as they are usually too dynamic. These class
decorators can be used with native classes, however:
* ``mypy_extensions.trait`` (for defining :ref:`trait types <trait-types>`)
* ``mypy_extensions.mypyc_attr`` (see :ref:`above <inheritance>`)
* ``dataclasses.dataclass``
Dataclasses have partial native support, and they aren't as efficient
as pure native classes.
.. note::
If a class definition uses an unsupported class decorator, *mypyc
compiles the class into a regular Python class*.
Deleting attributes
-------------------
By default, attributes defined in native classes can't be deleted. You
can explicitly allow certain attributes to be deleted by using
``__deletable__``::
class Cls:
x: int = 0
y: int = 0
other: int = 0
__deletable__ = ['x', 'y'] # 'x' and 'y' can be deleted
o = Cls()
del o.x # OK
del o.y # OK
del o.other # Error
You must initialize the ``__deletable__`` attribute in the class body,
using a list or a tuple expression with only string literal items that
refer to attributes. These are not valid::
a = ['x', 'y']
class Cls:
x: int
y: int
__deletable__ = a # Error: cannot use variable 'a'
__deletable__ = ('a',) # Error: not in a class body
Other properties
----------------
Instances of native classes don't usually have a ``__dict__`` attribute.

View file

@ -0,0 +1,54 @@
Miscellaneous native operations
===============================
This is a list of various non-type-specific operations that have
custom native implementations. If an operation has no native
implementation, mypyc will use fallback generic implementations that
are often not as fast.
.. note::
Operations specific to various primitive types are described
in the following sections.
Operators
---------
* ``x is y`` (this is very fast for all types)
Functions
---------
* ``isinstance(obj, type: type)``
* ``isinstance(obj, type: tuple)``
* ``cast(<type>, obj)``
* ``type(obj)``
* ``len(obj)``
* ``id(obj)``
* ``iter(obj)``
* ``next(iter: Iterator)``
* ``hash(obj)``
* ``getattr(obj, attr)``
* ``getattr(obj, attr, default)``
* ``setattr(obj, attr, value)``
* ``hasattr(obj, attr)``
* ``delattr(obj, name)``
* ``slice(start, stop, step)``
* ``globals()``
Method decorators
-----------------
* ``@property``
* ``@staticmethod``
* ``@classmethod``
* ``@abc.abstractmethod``
Statements
----------
These variants of statements have custom implementations:
* ``for ... in seq:`` (for loop over a sequence)
* ``for ... in enumerate(...):``
* ``for ... in zip(...):``

View file

@ -0,0 +1,244 @@
.. _performance-tips:
Performance tips and tricks
===========================
Performance optimization is part art, part science. Just using mypyc
in a simple manner will likely make your code faster, but squeezing
the most performance out of your code requires the use of some
techniques we'll summarize below.
Profiling
---------
If you are speeding up existing code, understanding where time is
spent is important. Mypyc speeds up code that you compile. If most of
the time is spent elsewhere, you may come back disappointed. For
example, if you spend 40% of time outside compiled code, even if
compiled code would go 100x faster, overall performance will only be
2.5x faster.
A simple (but often effective) approach is to record the time in
various points of program execution using ``time.time()``, and to
print out elapsed time (or to write it to a log file).
The stdlib modules ``profile`` and ``cProfile`` can provide much more
detailed data. (But these only work well with non-compiled code.)
Avoiding slow libraries
-----------------------
If profiling indicates that a lot of time is spent in the stdlib or
third-party libraries, you still have several options.
First, if most time is spent in a few library features, you can
perhaps easily reimplement them in type-annotated Python, or extract
the relevant code and annotate it. Now it may be easy to compile this
code to speed it up.
Second, you may be able to avoid the library altogether, or use an
alternative, more efficient library to achieve the same purpose.
Type annotations
----------------
As discussed earlier, type annotations are key to major performance
gains. You should at least consider adding annotations to any
performance-critical functions and classes. It may also be helpful to
annotate code called by this code, even if it's not compiled, since
this may help mypy infer better types in the compile code. If you use
libraries, ensure they have stub files with decent type annotation
coverage. Writing a stub file is often easy, and you only need to
annotate features you use a lot.
If annotating external code or writing stubs feel too burdensome, a
simple workaround is to annotate variables explicitly. For example,
here we call ``acme.get_items()``, but it has no type annotation. We
can use an explicit type annotation for the variable to which we
assign the result::
from typing import List, Tuple
import acme
def work() -> None:
# Annotate "items" to help mypyc
items: List[Tuple[int, str]] = acme.get_items()
for item in items:
... # Do some work here
Without the annotation on ``items``, the type would be ``Any`` (since
``acme`` has no type annotation), resulting in slower, generic
operations being used later in the function.
Avoiding slow Python features
-----------------------------
Mypyc can optimize some features more effectively than others. Here
the difference is sometimes big -- some things only get marginally
faster at best, while others can get 10x faster, or more. Avoiding
these slow features in performance-critical parts of your code can
help a lot.
These are some of the most important things to avoid:
* Using class decorators or metaclasses in compiled code (that aren't
properly supported by mypyc)
* Heavy reliance on interpreted Python libraries (C extensions are
usually fine)
These things also tend to be relatively slow:
* Using Python classes and instances of Python classes (native classes
are much faster)
* Calling decorated functions (``@property``, ``@staticmethod``, and
``@classmethod`` are special cased and thus fast)
* Calling nested functions
* Calling functions or methods defined in other compilation units
* Using ``*args`` or ``**kwargs``
* Using generator functions
* Using floating point numbers (they are relatively unoptimized)
* Using callable values (i.e. not leveraging early binding to call
functions or methods)
Nested functions can often be replaced with module-level functions or
methods of native classes.
Callable values and nested functions can sometimes be replaced with an
instance of a native class with a single method only, such as
``call(...)``. You can derive the class from an ABC, if there are
multiple possible functions.
.. note::
Some slow features will likely get efficient implementations in the
future. You should check this section every once in a while to see
if some additional operations are fast.
Using fast native features
--------------------------
Some native operations are particularly quick relative to the
corresponding interpreted operations. Using them as much as possible
may allow you to see 10x or more in performance gains.
Some things are not much (or any) faster in compiled code, such as set
math operations. In contrast, calling a method of a native class is
much faster in compiled code.
If you are used to optimizing for CPython, you might have replaced
some class instances with dictionaries, as they can be
faster. However, in compiled code, this "optimization" would likely
slow down your code.
Similarly, caching a frequently called method in a local variable can
help in CPython, but it can slow things down in compiled code, since
the code won't use :ref:`early binding <early-binding>`::
def squares(n: int) -> List[int]:
a = []
append = a.append # Not a good idea in compiled code!
for i in range(n):
append(i * i)
return a
Here are examples of features that are fast, in no particular order
(this list is *not* exhaustive):
* Calling compiled functions directly defined in the same compilation
unit (with positional and/or keyword arguments)
* Calling methods of native classes defined in the same compilation
unit (with positional and/or keyword arguments)
* Many integer operations
* Booleans
* :ref:`Native list operations <list-ops>`, such as indexing,
``append``, and list comprehensions
* While loops
* For loops over ranges and lists, and with ``enumerate`` or ``zip``
* Reading dictionary items
* ``isinstance()`` checks against native classes and instances of
primitive types (and unions of them)
* Accessing local variables
* Accessing attributes of native classes
* Accessing final module-level attributes
* Comparing strings for equality
These features are also fast, but somewhat less so (relative to other
related operations):
* Constructing instances of native classes
* Constructing dictionaries
* Setting dictionary items
* Native :ref:`dict <dict-ops>` and :ref:`set <set-ops>` operations
* Accessing module-level variables
Generally anything documented as a native operation is fast, even if
it's not explicitly mentioned here
Adjusting garbage collection
----------------------------
Compilation does not speed up cyclic garbage collection. If everything
else gets much faster, it's possible that garbage collection will take
a big fraction of time. You can use ``gc.set_threshold()`` to adjust
the garbage collector to run less often::
import gc
# Spend less time in gc; do this before significant computation
gc.set_threshold(150000)
... # Actual work happens here
Fast interpreter shutdown
-------------------------
If you allocate many objects, it's possible that your program spends a
lot of time cleaning up when the Python runtime shuts down. Mypyc
won't speed up the shutdown of a Python process much.
You can call ``os._exit(code)`` to immediately terminate the Python
process, skipping normal cleanup. This can give a nice boost to a
batch process or a command-line tool.
.. note::
This can be dangerous and can lose data. You need to ensure
that all streams are flushed and everything is otherwise cleaned up
properly.
Work smarter
------------
Usually there are many things you can do to improve performance, even
if most tweaks will yield only minor gains. The key to being effective
is to focus on things that give a large gain with a small effort.
For example, low-level optimizations, such as avoiding a nested
function, can be pointless, if you could instead avoid a metaclass --
to allow a key class to be compiled as a native class. The latter
optimization could speed up numerous method calls and attribute
accesses, just like that.

View file

@ -0,0 +1,47 @@
.. _set-ops:
Native set operations
======================
These ``set`` operations have fast, optimized implementations. Other
set operations use generic implementations that are often slower.
Construction
------------
Construct set with specific items:
* ``{item0, ..., itemN}``
Construct empty set:
* ``set()``
Construct set from iterable:
* ``set(x: Iterable)``
Set comprehensions:
* ``{... for ... in ...}``
* ``{... for ... in ... if ...}``
Operators
---------
* ``item in s``
Methods
-------
* ``s.add(item)``
* ``s.remove(item)``
* ``s.discard(item)``
* ``s.update(x: Iterable)``
* ``s.clear()``
* ``s.pop()``
Functions
---------
* ``len(s: set)``

View file

@ -0,0 +1,35 @@
.. _str-ops:
Native string operations
========================
These ``str`` operations have fast, optimized implementations. Other
string operations use generic implementations that are often slower.
Construction
------------
* String literal
* ``str(x: int)``
* ``str(x: object)``
Operators
---------
* Concatenation (``s1 + s2``)
* Indexing (``s[n]``)
* Slicing (``s[n:m]``, ``s[n:]``, ``s[:m]``)
* Comparisons (``==``, ``!=``)
* Augmented assignment (``s1 += s2``)
Methods
-------
* ``s1.endswith(s2: str)``
* ``s.join(x: Iterable)``
* ``s.replace(old: str, new: str)``
* ``s.replace(old: str, new: str, count: int)``
* ``s.split()``
* ``s.split(sep: str)``
* ``s.split(sep: str, maxsplit: int)``
* ``s1.startswith(s2: str)``

View file

@ -0,0 +1,33 @@
.. _tuple-ops:
Native tuple operations
=======================
These ``tuple`` operations have fast, optimized implementations. Other
tuple operations use generic implementations that are often slower.
Unless mentioned otherwise, these operations apply to both fixed-length
tuples and variable-length tuples.
Construction
------------
* ``item0, ..., itemN`` (construct a tuple)
* ``tuple(lst: list)`` (construct a variable-length tuple)
* ``tuple(lst: Iterable)`` (construct a variable-length tuple)
Operators
---------
* ``tup[n]`` (integer index)
* ``tup[n:m]``, ``tup[n:]``, ``tup[:m]`` (slicing)
Statements
----------
* ``item0, ..., itemN = tup`` (for fixed-length tuples)
Functions
---------
* ``len(tup: tuple)``

View file

@ -0,0 +1,314 @@
.. _using-type-annotations:
Using type annotations
======================
You will get the most out of mypyc if you compile code with precise
type annotations. Not all type annotations will help performance
equally, however. Using types such as :ref:`primitive types
<primitive-types>`, :ref:`native classes <native-class-intro>`,
:ref:`union types <union-types>`, :ref:`trait types <trait-types>`,
and :ref:`tuple types <tuple-types>` as much as possible is a key to
major performance gains over CPython.
In contrast, some other types, including ``Any``, are treated as
:ref:`erased types <erased-types>`. Operations on erased types use
generic operations that work with arbitrary objects, similar to how
the CPython interpreter works. If you only use erased types, the only
notable benefits over CPython will be the removal of interpreter
overhead (from compilation) and a bit of :ref:`early binding
<early-binding>`, which will usually only give minor performance
gains.
.. _primitive-types:
Primitive types
---------------
The following built-in types are treated as *primitive types* by
mypyc, and many operations on these types have efficient
implementations:
* ``int`` (:ref:`native operations <int-ops>`)
* ``float`` (:ref:`native operations <float-ops>`)
* ``bool`` (:ref:`native operations <bool-ops>`)
* ``str`` (:ref:`native operations <str-ops>`)
* ``List[T]`` (:ref:`native operations <list-ops>`)
* ``Dict[K, V]`` (:ref:`native operations <dict-ops>`)
* ``Set[T]`` (:ref:`native operations <set-ops>`)
* ``Tuple[T, ...]`` (variable-length tuple; :ref:`native operations <tuple-ops>`)
* ``None``
The link after each type lists all supported native, optimized
operations for the type. You can use all operations supported by
Python, but *native operations* will have custom, optimized
implementations.
Primitive containers
--------------------
Primitive container objects such as ``list`` and ``dict`` don't
maintain knowledge of the item types at runtime -- the item type is
*erased*.
This means that item types are checked when items are accessed, not
when a container is passed as an argument or assigned to another
variable. For example, here we have a runtime type error on the final
line of ``example`` (the ``Any`` type means an arbitrary, unchecked
value)::
from typing import List, Any
def example(a: List[Any]) -> None:
b: List[int] = a # No error -- items are not checked
print(b[0]) # Error here -- got str, but expected int
example(["x"])
.. _native-class-intro:
Native classes
--------------
Classes that get compiled to C extensions are called native
classes. Most common operations on instances of these classes are
optimized, including construction, attribute access and method calls.
Native class definitions look exactly like normal Python class
definitions. A class is usually native if it's in a compiled module
(though there are some exceptions).
Consider this example:
.. code-block::
class Point:
def __init__(self, x: int, y: int) -> None:
self.x = x
self.y = y
def shift(p: Point) -> Point:
return Point(p.x + 1, p.y + 1)
All operations in the above example use native operations, if the file
is compiled.
Native classes have some notable different from Python classes:
* Only attributes and methods defined in the class body or methods are
supported. If you try to assign to an undefined attribute outside
the class definition, ``AttributeError`` will be raised. This enables
an efficient memory layout and fast method calls for native classes.
* Native classes usually don't define the ``__dict__`` attribute (they
don't have an attribute dictionary). This follows from only having
a specific set of attributes.
* Native classes can't have an arbitrary metaclass or use most class
decorators.
Native classes only support single inheritance. A limited form of
multiple inheritance is supported through *trait types*. You generally
must inherit from another native class (or ``object``). By default,
you can't inherit a Python class from a native class (but there's
an :ref:`override <inheritance>` to allow that).
See :ref:`native-classes` for more details.
.. _tuple-types:
Tuple types
-----------
Fixed-length
`tuple types <https://mypy.readthedocs.io/en/stable/kinds_of_types.html#tuple-types>`_
such as ``Tuple[int, str]`` are represented
as :ref:`value types <value-and-heap-types>` when stored in variables,
passed as arguments, or returned from functions. Value types are
allocated in the low-level machine stack or in CPU registers, as
opposed to *heap types*, which are allocated dynamically from the
heap.
Like all value types, tuples will be *boxed*, i.e. converted to
corresponding heap types, when stored in Python containers, or passed
to non-native code. A boxed tuple value will be a regular Python tuple
object.
.. _union-types:
Union types
-----------
`Union types <https://mypy.readthedocs.io/en/stable/kinds_of_types.html#union-types>`_
and
`optional types <https://mypy.readthedocs.io/en/stable/kinds_of_types.html#optional-types-and-the-none-type>`_
that contain primitive types, native class types and
trait types are also efficient. If a union type has
:ref:`erased <erased-types>` items, accessing items with
non-erased types is often still quite efficient.
A value with a union types is always :ref:`boxed <value-and-heap-types>`,
even if it contains a value that also has an unboxed representation, such
as an integer or a boolean.
For example, using ``Optional[int]`` is quite efficient, but the value
will always be boxed. A plain ``int`` value will usually be faster, since
it has an unboxed representation.
.. _trait-types:
Trait types
-----------
Trait types enable a form of multiple inheritance for native classes.
A native class can inherit any number of traits. Trait types are
defined as classes using the ``mypy_extensions.trait`` decorator::
from mypy_extensions import trait
@trait
class MyTrait:
def method(self) -> None:
...
Traits can define methods, properties and attributes. They often
define abstract methods. Traits can be generic.
If a class subclasses both a non-trait class and traits, the traits
must be placed at the end of the base class list::
class Base: ...
class Derived(Base, MyTrait, FooTrait): # OK
...
class Derived2(MyTrait, FooTrait, Base):
# Error: traits should come last
...
Traits have some special properties:
* You shouldn't create instances of traits (though mypyc does not
prevent it yet).
* Traits can subclass other traits, but they can't subclass non-trait
classes (other than ``object``).
* Accessing methods or attributes through a trait type is somewhat
less efficient than through a native class type, but this is much
faster than through Python class types or other
:ref:`erased types <erased-types>`.
You need to install ``mypy-extensions`` to use ``@trait``:
.. code-block:: text
pip install --upgrade mypy-extensions
.. _erased-types:
Erased types
------------
Mypyc supports many other kinds of types as well, beyond those
described above. However, these types don't have customized
operations, and they are implemented using *type erasure*. Type
erasure means that all other types are equivalent to untyped values at
runtime, i.e. they are the equivalent of the type ``Any``. Erased
types include these:
* Python classes (including ABCs)
* Non-mypyc extension types and primitive types (including built-in
types that are not primitives)
* `Callable types <https://mypy.readthedocs.io/en/stable/kinds_of_types.html#callable-types-and-lambdas>`_
* `Type variable types <https://mypy.readthedocs.io/en/stable/generics.html>`_
* Type `Any <https://mypy.readthedocs.io/en/stable/dynamic_typing.html>`_
* Protocol types
Using erased types can still improve performance, since they can
enable better types to be inferred for expressions that use these
types. For example, a value with type ``Callable[[], int]`` will not
allow native calls. However, the return type is a primitive type, and
we can use fast operations on the return value::
from typing import Callable
def call_and_inc(f: Callable[[], int]) -> int:
# Slow call, since f has an erased type
n = f()
# Fast increment; inferred type of n is int (primitive type)
n += 1
return n
If the type of the argument ``f`` was ``Any``, the type of ``n`` would
also be ``Any``, resulting in a generic, slower increment operation
being used.
Strict runtime type checking
----------------------------
Compiled code ensures that any variable or expression with a
non-erased type only has compatible values at runtime. This is in
contrast with using *optional static typing*, such as by using mypy,
when type annotations are not enforced at runtime. Mypyc ensures
type safety both statically and at runtime.
``Any`` types and erased types in general can compromise type safety,
and this is by design. Inserting strict runtime type checks for all
possible values would be too expensive and against the goal of
high performance.
.. _value-and-heap-types:
Value and heap types
--------------------
In CPython, memory for all objects is dynamically allocated on the
heap. All Python types are thus *heap types*. In compiled code, some
types are *value types* -- no object is (necessarily) allocated on the
heap. ``bool``, ``None`` and fixed-length tuples are value types.
``int`` is a hybrid. For typical integer values, it is a value
type. Large enough integer values, those that require more than 63
bits (or 31 bits on 32-bit platforms) to represent, use a heap-based
representation (same as CPython).
Value types have a few differences from heap types:
* When an instance of a value type is used in a context that expects a
heap value, for example as a list item, it will transparently switch
to a heap-based representation (boxing) as needed.
* Similarly, mypyc transparently changes from a heap-based
representation to a value representation (unboxing).
* Object identity of integers and tuples is not preserved. You should
use ``==`` instead of ``is`` if you are comparing two integers or
fixed-length tuples.
* When an instance of a subclass of a value type is converted to the
base type, it is implicitly converted to an instance of the target
type. For example, a ``bool`` value assigned to a variable with an
``int`` type will be converted to the corresponding integer.
The latter conversion is the only implicit type conversion that
happens in mypyc programs.
Example::
def example() -> None:
# A small integer uses the value (unboxed) representation
x = 5
# A large integer the the heap (boxed) representation
x = 2**500
# Lists always contain boxed integers
a = [55]
# When reading from a list, the object is automatically unboxed
x = a[0]
# True is converted to 1 on assignment
x = True