Modules and Packaging

At some point, you will want to organize and distribute your code library for the whole world to share, preferably on PyPI so that it is pip installable.

References

This notebook shows a bare-bones version of creating and distributing a project to PyPI. Please follow the instructions in the official documentations. For convenience, you can use the sample project as a template.

• Packaging and Distributing Projects
• A sample Python project

For more about how to organize the structure of your package

• Official tutorial on packages

If you are still confused about what __init__.py does, this blog post might help.

Install packages we will use for packaging

! pip install -U pip
! pip install twine

Requirement already up-to-date: pip in /usr/local/lib/python2.7/site-packages
Requirement already satisfied: twine in /usr/local/lib/python2.7/site-packages
Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python2.7/site-packages (from twine)
Requirement already satisfied: pkginfo>=1.4.2 in /usr/local/lib/python2.7/site-packages (from twine)
Requirement already satisfied: requests-toolbelt>=0.8.0 in /usr/local/lib/python2.7/site-packages (from twine)
Requirement already satisfied: tqdm>=4.14 in /usr/local/lib/python2.7/site-packages (from twine)
Requirement already satisfied: requests!=2.15,!=2.16,>=2.5.0 in /usr/local/lib/python2.7/site-packages (from twine)

Modules

In Pythoh, any .py file is a module in that it can be imported. Because the interpreter runs the entrie file when a moudle is imported, it is traditional to use a guard to ignore code that should only run when the file is executed as a script.

%%file foo.py
"""
When this file is imported with `import foo`,
only `useful_func1()` and `useful_func()` are loaded,
and the test code `assert ...` is ignored. However,
when we run foo.py as a script `python foo.py`, then
the two assert statements are run.
Most commonly, the code under `if __naem__ == '__main__':`
consists of simple examples or test cases for the functions
defined in the moule.
"""

def useful_func1():
    pass

def useful_fucn2():
    pass

if __name__ == '__main__':
    assert(useful_func1() is None)
    assert(useful_fucn2() is None)

Overwriting foo.py

Organization of files in a module

When the number of files you write grow large, you will probably want to orgnize them into their own directory structure. To make a folder a module, you just need to include a file named __init__.py in the folder. This file can be empty. For example, here is a module named pkg with sub-modules sub1 and sub2.

./pkg:
__init__.py foo.py      sub1        sub2

./pkg/sub1:
__init__.py     more_sub1_stuff.py  sub1_stuff.py

./pkg/sub2:
__init__.py sub2_stuff.py

import pkg.foo as foo

foo.f1()

1

import pkg

pkg.foo.f1()

1

How to import a module at the same level

Within a package, we need to use absolute path names for importing other modules in the same directory. This prevents confusion as to whether you want to import a system moudle with the same name. For example, foo.sub1.more_sub1_stuff.py imports functions from foo.sub1.sub1_stuff.py

! cat pkg/sub1/more_sub1_stuff.py

from pkg.sub1.sub1_stuff import g1, g2

def g3():
    return 'g3 uses %s, %s' % (g1(), g2())

from pkg.sub1.more_sub1_stuff import g3

g3()

'g3 uses g1, g2'

How to import a moudle at a different level

Again, just use absolute paths. For example, sub2_stuff.py in the sub2 directory uses functions from sub1_stuff.py in the sub1 directory:

! cat pkg/sub2/sub2_stuff.py

from pkg.sub1.sub1_stuff import g1, g2

def h1():
    return g1()

def h2():
    return g1() + g2()

from pkg.sub2.sub2_stuff import h2

h2()

'g1g2'

Distributing your package

Suppose we want to distribute our code as a library (for example, on PyPI so that it cnn be installed with pip). Let’s create an sta663-<username> (the username part is just to avoid name conflicts) library containing the pkg package and some other files:

• README.md: some information about the library
• sta663.py: a standalone module
• run_sta663.py: a script (intended for use as python run_sta663.py)

! ls -R sta663

MANIFEST                dist                    sta663.py
README.txt              pkg                     sta663_cliburn.egg-info
__init__.py             run_sta663.py           sub1
__pycache__             setup.py                sub2
build                   sta663.egg-info

sta663/__pycache__:
__init__.cpython-36.pyc sta663.cpython-36.pyc

sta663/build:
bdist.macosx-10.12-x86_64 lib                       scripts-3.6

sta663/build/bdist.macosx-10.12-x86_64:

sta663/build/lib:
pkg       sta663.py

sta663/build/lib/pkg:
__init__.py foo.py      sub1        sub2

sta663/build/lib/pkg/sub1:
__init__.py        more_sub1_stuff.py sub1_stuff.py

sta663/build/lib/pkg/sub2:
__init__.py   sub2_stuff.py

sta663/build/scripts-3.6:
run_sta663.py

sta663/dist:
sta663_cliburn-1.0-py3.6.egg

sta663/pkg:
__init__.py __pycache__ foo.py      sub1        sub2

sta663/pkg/__pycache__:
__init__.cpython-36.pyc foo.cpython-36.pyc

sta663/pkg/sub1:
__init__.py        __pycache__        sub1_stuff.py
__init__.py~       more_sub1_stuff.py

sta663/pkg/sub1/__pycache__:
__init__.cpython-36.pyc   sub1_stuff.cpython-36.pyc

sta663/pkg/sub2:
__init__.py   __init__.py~  sub2_stuff.py

sta663/sta663.egg-info:
PKG-INFO             dependency_links.txt
SOURCES.txt          top_level.txt

sta663/sta663_cliburn.egg-info:
PKG-INFO             dependency_links.txt
SOURCES.txt          top_level.txt

sta663/sub1:
__intit__.py  __intit__.py~

sta663/sub2:
__iniit__.py  __iniit__.py~

! cat sta663/run_sta663.py

import pkg.foo as foo
from pkg.sub1.more_sub1_stuff import g3
from pkg.sub2.sub2_stuff import h2

print foo.f1()
print g3()
print h2()

Using distutils

All we need to do is to write a setup.py file.

%%file sta663/setup.py
from setuptools import setup

setup(name = "sta663-cliburn",
      version = "1.0",
      author='Cliburn Chan',
      author_email='cliburn.chan@duke.edu',
      url='http://people.duke.edu/~ccc14/sta-663-2018/',
      py_modules = ['sta663'],
      packages=setuptools.find_packages(),
      scripts = ['run_sta663.py'],
      python_requires='>=3',
      )

Overwriting sta663/setup.py

Build a source archive for distribution

%%bash

cd sta663
python setup.py sdist
cd -

/Users/cliburn/_teach/sta-663-2018/project/Packaging

Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    packages=setuptools.find_packages(),
NameError: name 'setuptools' is not defined

! ls -R sta663

MANIFEST                dist                    sta663.py
README.txt              pkg                     sta663_cliburn.egg-info
__init__.py             run_sta663.py           sub1
__pycache__             setup.py                sub2
build                   sta663.egg-info

sta663/__pycache__:
__init__.cpython-36.pyc sta663.cpython-36.pyc

sta663/build:
bdist.macosx-10.12-x86_64 lib                       scripts-3.6

sta663/build/bdist.macosx-10.12-x86_64:

sta663/build/lib:
pkg       sta663.py

sta663/build/lib/pkg:
__init__.py foo.py      sub1        sub2

sta663/build/lib/pkg/sub1:
__init__.py        more_sub1_stuff.py sub1_stuff.py

sta663/build/lib/pkg/sub2:
__init__.py   sub2_stuff.py

sta663/build/scripts-3.6:
run_sta663.py

sta663/dist:
sta663_cliburn-1.0-py3.6.egg

sta663/pkg:
__init__.py __pycache__ foo.py      sub1        sub2

sta663/pkg/__pycache__:
__init__.cpython-36.pyc foo.cpython-36.pyc

sta663/pkg/sub1:
__init__.py        __pycache__        sub1_stuff.py
__init__.py~       more_sub1_stuff.py

sta663/pkg/sub1/__pycache__:
__init__.cpython-36.pyc   sub1_stuff.cpython-36.pyc

sta663/pkg/sub2:
__init__.py   __init__.py~  sub2_stuff.py

sta663/sta663.egg-info:
PKG-INFO             dependency_links.txt
SOURCES.txt          top_level.txt

sta663/sta663_cliburn.egg-info:
PKG-INFO             dependency_links.txt
SOURCES.txt          top_level.txt

sta663/sub1:
__intit__.py  __intit__.py~

sta663/sub2:
__iniit__.py  __iniit__.py~

Distribution

You can now distribute sta663-1.0.tar.gz to somebody else for installation in the usual way.

%%bash

cp sta663/dist/sta663_cliburn-1.0-py3.6.egg /tmp
cd /tmp
tar xzf sta663_cliburn-1.0-py3.6.egg
cd sta663-1.0
python setup.py install

running install
running bdist_egg
running egg_info
writing sta663.egg-info/PKG-INFO
writing top-level names to sta663.egg-info/top_level.txt
writing dependency_links to sta663.egg-info/dependency_links.txt
reading manifest file 'sta663.egg-info/SOURCES.txt'
writing manifest file 'sta663.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.12-x86_64/egg
running install_lib
running build_py
creating build/bdist.macosx-10.12-x86_64/egg
copying build/lib/sta663.py -> build/bdist.macosx-10.12-x86_64/egg
creating build/bdist.macosx-10.12-x86_64/egg/pkg
creating build/bdist.macosx-10.12-x86_64/egg/pkg/sub1
copying build/lib/pkg/sub1/__init__.py -> build/bdist.macosx-10.12-x86_64/egg/pkg/sub1
copying build/lib/pkg/sub1/more_sub1_stuff.py -> build/bdist.macosx-10.12-x86_64/egg/pkg/sub1
copying build/lib/pkg/sub1/sub1_stuff.py -> build/bdist.macosx-10.12-x86_64/egg/pkg/sub1
copying build/lib/pkg/__init__.py -> build/bdist.macosx-10.12-x86_64/egg/pkg
creating build/bdist.macosx-10.12-x86_64/egg/pkg/sub2
copying build/lib/pkg/sub2/__init__.py -> build/bdist.macosx-10.12-x86_64/egg/pkg/sub2
copying build/lib/pkg/sub2/sub2_stuff.py -> build/bdist.macosx-10.12-x86_64/egg/pkg/sub2
copying build/lib/pkg/foo.py -> build/bdist.macosx-10.12-x86_64/egg/pkg
byte-compiling build/bdist.macosx-10.12-x86_64/egg/sta663.py to sta663.pyc
byte-compiling build/bdist.macosx-10.12-x86_64/egg/pkg/sub1/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.12-x86_64/egg/pkg/sub1/more_sub1_stuff.py to more_sub1_stuff.pyc
byte-compiling build/bdist.macosx-10.12-x86_64/egg/pkg/sub1/sub1_stuff.py to sub1_stuff.pyc
byte-compiling build/bdist.macosx-10.12-x86_64/egg/pkg/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.12-x86_64/egg/pkg/sub2/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.12-x86_64/egg/pkg/sub2/sub2_stuff.py to sub2_stuff.pyc
byte-compiling build/bdist.macosx-10.12-x86_64/egg/pkg/foo.py to foo.pyc
creating build/bdist.macosx-10.12-x86_64/egg/EGG-INFO
installing scripts to build/bdist.macosx-10.12-x86_64/egg/EGG-INFO/scripts
running install_scripts
running build_scripts
creating build/bdist.macosx-10.12-x86_64/egg/EGG-INFO/scripts
copying build/scripts-2.7/run_sta663.py -> build/bdist.macosx-10.12-x86_64/egg/EGG-INFO/scripts
changing mode of build/bdist.macosx-10.12-x86_64/egg/EGG-INFO/scripts/run_sta663.py to 755
copying sta663.egg-info/PKG-INFO -> build/bdist.macosx-10.12-x86_64/egg/EGG-INFO
copying sta663.egg-info/SOURCES.txt -> build/bdist.macosx-10.12-x86_64/egg/EGG-INFO
copying sta663.egg-info/dependency_links.txt -> build/bdist.macosx-10.12-x86_64/egg/EGG-INFO
copying sta663.egg-info/top_level.txt -> build/bdist.macosx-10.12-x86_64/egg/EGG-INFO
creating 'dist/sta663-1.0-py2.7.egg' and adding 'build/bdist.macosx-10.12-x86_64/egg' to it
removing 'build/bdist.macosx-10.12-x86_64/egg' (and everything under it)
Processing sta663-1.0-py2.7.egg
Removing /usr/local/lib/python2.7/site-packages/sta663-1.0-py2.7.egg
Copying sta663-1.0-py2.7.egg to /usr/local/lib/python2.7/site-packages
sta663 1.0 is already the active version in easy-install.pth
Installing run_sta663.py script to /usr/local/bin

Installed /usr/local/lib/python2.7/site-packages/sta663-1.0-py2.7.egg
Processing dependencies for sta663==1.0
Finished processing dependencies for sta663==1.0

zip_safe flag not set; analyzing archive contents...

See __init__.py to understand what can be imported from where

! cat sta663/__init__.py

import pkg
from pkg.sub1.sub1_stuff import g1

! cat sta663/pkg/__init__.py

! cat sta663/pkg/sub1/__init__.py

from sta663.pkg.sub1.sub1_stuff import g1, g2
from sta663.pkg.sub1.more_sub1_stuff import g3

! cat sta663/pkg/sub2/__init__.py

from sta663.pkg.sub2.sub2_stuff import h1, h2

Using imports from the sta663 package

import sta663

[x for x in dir(sta663) if not x.startswith('__')]

['g1', 'pkg']

sta663.g1()

'g1'

from sta663 import pkg

[x for x in dir(pkg) if not x.startswith('__')]

['sub1', 'sub2']

from sta663.pkg import sub1, sub2

[x for x in dir(sub1) if not x.startswith('__')]

['g1', 'g2', 'g3', 'more_sub1_stuff', 'sub1_stuff']

[x for x in dir(sub2) if not x.startswith('__')]

['h1', 'h2', 'sub2_stuff']

sub1.g1(), sub1.g2(), sub1.g3()

('g1', 'g2', 'g3 uses g1, g2')

sub2.h1(), sub2.h2()

('g1', 'g1g2')

sta663.pkg.sub2.sub2_stuff.h1()

'g1'

sta663.pkg.sub2.sub2_stuff.h2()

'g1g2'

Distributing to PyPI

For testing, please upload to TestPyPI which is cleaned on a regular basis. See instructions at https://packaging.python.org/guides/using-testpypi/#using-test-pypi

• Note 1: You need to confirm your email address after registration.
• Note 2: You can easily delete any uploaded packages by logging in at https://test.pypi.org.

When your package is ready for public release, you can upload to PyPI. See instructions at https://packaging.python.org/tutorials/distributing-packages/#id78

%%bash

export TWINE_USERNAME='cliburn'
export TWINE_PASSWORD=''
twine upload --repository-url https://test.pypi.org/legacy/ sta663/dist/*

Uploading distributions to https://test.pypi.org/legacy/
Uploading sta663_cliburn-1.0-py3.6.egg
100%|##########| 7.75K/7.75K [00:00<00:00, 61.9KB/s]

%%bash

pip install --index-url https://test.pypi.org/simple/ sta663

Requirement already satisfied: sta663 in /usr/local/lib/python2.7/site-packages/sta663-1.0-py2.7.egg

sta663 requires Python '>=3' but the running Python is 2.7.13