Python for System Administrators

# Python for System Administrators

Sean Raven

2018-03-24

---

# Installing Python

---

## Installing on Windows

Official website: https://www.python.org/

Anaconda (recommended): https://www.anaconda.com/

During install, look for and enable the option `Add Python to System PATH`.

If using Anaconda, after install, create a shortcut to launch IDLE:
1. Open the install folder (probably `C:\Anaconda`)
2. Right-click on `pythonw.exe`
    - Select `Send To` > `Desktop (Create Shortcut)`
3. Right-click on the shortcut and select `Properties`
    1. Go to the shortcut tab
    2. Add ` -m idlelib` to the end of Target
        - It should look something like `"C:\...\pythonw.exe" -m idlelib`

---

## Installing on Linux or MacOS

You probably already have Python installed.
Try opening a Terminal and type `python3`.
You should see a message like `Python 3. ...` followed by a prompt `>>> `.
Press `CTRL-D` to close it.

Also check that you have IDLE.
In the terminal, run `idle3`.
On some Linux distros, IDLE might be a separate package.
Install it with something like `apt install idle3`.
This will also add a launcher icon to your application menu.

On MacOS, if you want a launcher icon, you must download Python from https://www.python.org/ and install it.
This means you may have two different versions of Python installed.
This shouldn't be an issue.

---

# A Brief History of UNIX Scripting

---

## Bash/Pipes

Bash is the standard shell on most UNIX-based systems (Linux, MacOS, etc).
The user can type the name of a program and Bash will execute it.
Programs can accept additional options on the command line.
Programs have "standard" text streams, one for data input, one for data output, and one for error output.
The standard output of one command can be redirected to the standard input of another command.
This allows the user to create "pipelines" for processing text.

This system encouraged the creation of many small programs to "do one thing well" so they could be combined in pipelines.
One such program is `grep`, which prints lines that match a regular expression pattern.

`cat /usr/share/dict/words | grep '\<c...h\>'`

---

## Sed

Sed is the stream editor.
Given editing commands on the command line, it reads standard input, performs the editing commands, ant writes the result to standard output.
The most common command is `substitute` (abbr. `s`),
which performs a regular expression find-and-replace.

`cat /etc/passwd | sed 's/^$[^:]*$:.*:$[^:]*$$/\1 uses shell \2\t(&)/'`

---

## Awk

Named after the creators: Alfred Aho, Peter Weinberger, and Brian Kernighan of Bell Laboratories.
Awk creates a C-like programming language for processing text.
Awk reads standard input and separates it into records and fields.
For each record, it then executes the script, supplying the fields using numbered variables.

`df -h | awk '/dev\/hd/ { print $6 "\t: " $5 }'`

---

## Problems

- Awk always reads standard input and separates it into records and fields; there's nothing your code can do to prevent this.
- Bash's main purpose is to run external programs. These programs must be installed.

---

## Perl
#### The Swiss-Army Chainsaw

- Created by Larry Wall in 1987
- General-Purpose language
  - Only does exactly what your code says; nothing extra like Awk
  - Uses other code written in Perl; does not depend on external programs like Bash
- Built-in regular expression processing based on Grep and Sed.
- Built-in support for running external programs.

---

## Typical Perl code

```perl
@P=split//,".URRUU\c8R";@d=split//,"\nrekcah xinU / lreP rehtona tsuJ";sub p{
@p{"r$p","u$p"}=(P,P);pipe"r$p","u$p";++$p;($q*=2)+=$f=!fork;map{$P=$P[$f^ord
($p{$_})&6];$p{$_}=/ ^$P/ix?$P:close$_}keys%p}p;p;p;p;p;map{$p{$_}=~/^[P.]/&&
close$_}%p;wait until$?;map{/^r/&&<$_>}%p;$_=$d[$q];sleep rand(2)if/\S/;print
```

(not really)

(source: https://perl.plover.com/obfuscated/)

---

[How to Become a Hacker](http://catb.org/esr/faqs/hacker-howto.html)

---

# Basics

---

## A Basic Python Script

```python
print('Hello, world!')
name = input("What is your name? ")
print("Hello", name)
```

---

## Using the Python Shell

```python
$ python3
Python 3.4.3 (default, Nov 28 2017, 16:41:13) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = 4
>>> x = 'hi'
>>> x
'hi'
>>> x.upper()
'HI'
>>> x = 4
>>> x.upper()
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute 'upper'
>>> x = 'hi'
>>> type(x)
<class 'str'>
>>> dir(x)
[...]
>>> help(x.upper)
```

---

## Basic Types

- int
- float
- str

---

## String Literals

```python
print('This string can contain "double quotes" without escaping them.')
print("This string can contain 'single quotes' without escaping them.")
print('This string must \'escape\' single quotes.')

print( 'This is a newline: \n')
print(r'This is a backslash followed by n: \n')
print( 'A Windows filename: C:\\Users\\jstudent0\\data.txt')
print(r'A raw Windows filename: C:\Users\jstudent0\data.txt')
print( 'A regex to match a backslash: \\\\')
print(r'A raw regex to match a backslash: \\')

print('''This is a multiline string.

It can have ' and " without escaping, but can't have three ''s or "'s in a row.
''')
```

---

## More Strings

```python
>>> name = 'frederic'
>>> name[0]
'f'
>>> name[1:4]
'red'
>>> name[:-1]
'frederi'
>>> 'eric' in name
True
>>> s1 = 'hello'
>>> s2 = 'world'
>>> s3 = s1 + ' ' + s2
>>> s3
'hello world'
>>> s1 + 47
Traceback (most recent call last):
 File "<pyshell#63>", line 1, in <module>
 s1 + 47
TypeError: Can't convert 'int' object to str implicitly
>>> s1 + str(47)
'hello47'
>>> print(s3, "is", len(s3), "letters long")
hello world is 11 letters long
>>> '1,2,3'.split(',')
['1', '2', '3']
>>> ', '.join([1, 2, 3])
'1, 2, 3'
```

---

## String Formatting

```python
>>> import math
>>> "PI to 3 digits: {0:.3f}".format(math.pi)
'PI to 3 digits: 3.142'
>>> "Add zeros to ensure three digits after the point: {0:.03f}".format(math.pi)
'Add zeros to ensure three digits after the point: 3.142'
>>> "A large number with commas: {0:,}".format(10**7)
'A large number with commas: 10,000,000'
>>> expected = 5
>>> actual = 3
>>> "Display a percent: {0:.03%}".format(actual/expected)
'Display a percent: 60.000%'
>>> "A right-aligned field 20 characters wide: {0:>20s}".format("Hello")
'A right-aligned field 20 characters wide: Hello'
>>> "A left-aligned field 20 characters wide: {0:<20s}".format("Hello")
'A left-aligned field 20 characters wide: Hello '
>>> "A centered field 20 characters wide: {0:^20s}".format("Hello")
'A centered field 20 characters wide: Hello '
>>> "A field 20 characters wide, padded with stars: {0:*^20}".format("Hello")
'A field 20 characters wide, padded with stars: *******Hello********'
>>> "A field 20 characters wide, padded with pluses: {0:+^20}".format("Hello")
'A field 20 characters wide, padded with pluses: +++++++Hello++++++++'
```

---

## String Formatting

```python
>>> x, y = 3, 5
>>> "A point: ({0}, {1})".format(x, y)
'A point: (3, 5)'
>>> "A point with coordinates reversed: ({1}, {0})".format(y, x)
'A point with coordinates reversed: (3, 5)'
>>> "A repeated value: {0}{0}{0}".format("Hello")
'A repeated value: HelloHelloHello'
>>> "A hexidecimal number: {0:x}".format(127)
'A hexidecimal number: 7f'
>>> "A hexidecimal number, with 0x prefix: {0:#x}".format(127)
'A hexidecimal number, with 0x prefix: 0x7f'
>>> 
```

---

## Docstrings

If the first statement in a module (file), class, or function is a string literal,
it is automatically stored in a special attribute and referred to as a "documentation string" or docstring.
Docstrings are utilized by the `help` function and other tools, such as the `doctest` module.

```python
>>> def foo():
...     'this is a documentation string'
... 
>>> foo.__doc__
'this is a documentation string'
>>> help(foo)
Help on function foo in module __main__:

foo()
    this is a documentation string

```

---

## Docstrings

Programmers usually use triple-quoted strings so that they can span multiple lines

```python
>>> def bar():
        '''This is also a documentation string

It spans multiple lines to hold more information.
        It also has a doctest

>>> bar()
        >>>
        '''

>>> help(bar)
Help on function bar in module __main__:

bar()
    This is also a documentation string
    
    It spans multiple lines to hold more information.
    It also has a doctest

>>> bar()
    >>>

```

---

## Booleans

```python
>>> True
True
>>> False
False
>>> None
>>> print(None)
None
>>> x = 5
>>> 3 <= x < 7
True
>>> True and False
False
>>> True or False
True
>>> not False
True
>>> [1, 2, 3] == [1, 2, 3]
True
>>> [1, 2, 3] is [1, 2, 3]
False
>>> a = [1, 2, 3]
>>> b = a
>>> a is b
True
>>> sign = '+' if val >= 0 else '-'
```

---

## If Statements

```python
if condition:
    pass
elif condition:
    pass
else:
    pass
```

---

## While Statements

```python
while condition:
    pass
```

---

## Lists

```python
>>> myList = [1, 2, 3]
>>> myList.append(4)
>>> myList
[1, 2, 3, 4]
>>> myList[0]
1
>>> myList[-1]
4
>>> myList[1:3]
[2, 3]
>>> myList[::-1]
[4, 3, 2, 1]
>>> myList.pop()
4
>>> myList
[1, 2, 3]
>>> myList.pop(0)
1
>>> myList
[2, 3]
>>> 2 in myList
True
>>> 4 in myList
False
```

---

## Lists and References

```python
>>> myList = [1, 2, 3]
>>> myOtherList = myList
>>> myOtherList.append(4)
>>> myList
[1, 2, 3, 4]
>>> myCopiedList = myList[:]
>>> myCopiedList.append(5)
>>> myCopiedList
[1, 2, 3, 4, 5]
>>> myList
[1, 2, 3, 4]
>>> myOtherList
[1, 2, 3, 4]
```

---

## Tuples

```python
>>> point = (3, 5)
>>> point[0]
3
>>> point[-1]
5
>>> x, y = point
>>> x, y = 3, 5
>>> x, y = y, x
>>> x, y = y, x+y
>>> def make_point():
...     return 3, 5
... 
>>> point = make_point()
>>> x, y = point
>>> # or
>>> x, y = make_point()
```

---

## Sets

```python
>>> s = set()
>>> s.add(3)
>>> s.add(3)
>>> s
{3}
>>> l = [3]
>>> s = {3}
>>> 3 in l # this must compare 3 to every item in l
True
>>> 3 in s # this locates 3 using it's hashcode, and takes constant time
True
>>> a = set('abracadabra')
>>> b = set('alacazam')
>>> a      # unique letters in a
{'a', 'r', 'c', 'b', 'd'}
>>> b      # unique letters in b
{'a', 'z', 'c', 'l', 'm'}
>>> a - b  # set difference: letters in a but not in b
{'r', 'd', 'b'}
>>> a & b  # set intersection: letters in both a and b
{'a', 'c'}
>>> a | b  # set union: letters in either a or b
{'a', 'c', 'd', 'l', 'r', 'm', 'b', 'z'}
>>> a ^ b  # symmetric difference: letters in a or b but not both
{'d', 'l', 'r', 'm', 'b', 'z'}
>>> a ^ b == ( a - b ) | ( b - a )
True
```

---

## Dictionaries

```python
>>> tel = {'jack': 4098, 'sape': 4193}
>>> tel['guido'] = 4127
>>> tel
{'sape': 4193, 'jack': 4098, 'guido': 4127}
>>> tel['jack']
4098
>>> del tel['sape']
>>> tel['irv'] = 4127
>>> tel
{'irv': 4127, 'jack': 4098, 'guido': 4127}
>>> 'guido' in tel
True
>>> 'jack' not in tel
False
>>> points = {}
>>> points[3, 5] = 'floor' # use tuples (3, 5) and (4, 4) as dict keys
>>> points[4, 4] = 'wall'
```

---

## For Loops

`for` loops in Python can only be used with iterable types.
These are sometimes called "for-each" loops.
Most collection types are iterable.
Dictionaries have methods `keys`, `values`, and `items` that return iterators.
If an iterator returns a sequence, then the `for` loop can use sequence unpacking.

```python
>>> myList = [1, 2, 3, 4]
>>> for item in myList:
...     print(item)
... 
1
2
3
4
>>> for key, value in tel.items():
...     print(key, ':', value)
	
jack : 4098
irv : 4127
guido : 4127
```

---

## Functions

```python
def function(args):
    pass
```

By default, functions automatically return `None`

---

## Default Parameters

```python
>>> def foo(x=5):
...     print(x)
... 
>>> foo(3)
3
>>> foo()
5
```

---

## Keyword Arguments

```python
def hypotenuse(x, y):
    return (x**2 + y**2) ** 0.5

hypotenuse(3, 4)
hypotenuse(y=4, x=3)
```

---

## Arbitrary Argument Lists

```python
>>> def concat(*args, sep="/"):
...    return sep.join(args)
... 
>>> concat("earth", "mars", "venus")
'earth/mars/venus'
>>> concat("earth", "mars", "venus", sep=".")
'earth.mars.venus'
>>> def foo1(a, b, *, c='c', d='d', e='e'):
...     print('a', a, 'b', b, 'c', c, 'd', d, 'e', e)
... 
>>> foo1(1, 2, e=3)
a 1 b 2 c c d d e 3
>>> def foo2(a, b, **kwargs):
...     print('a', a, 'b', b, 'kwargs', kwargs)
... 
>>> foo2(a='bar', b='baz', c='blip', d='blop', e='henry')
a bar b baz kwargs {'e': 'henry', 'c': 'blip', 'd': 'blop'}
```

---

## Unpacking Argument Lists

```python
>>> def hypotenuse(x, y):
        return (x**2 + y**2) ** 0.5

>>> x = 3
>>> y = 4
>>> point = (x, y)
>>> hypotenuse(*point) # equivalent to hypotenuse(point[0], point[1])
5.0
>>> point = {'x': x, 'y': y}
>>> hypotenuse(**point) # equivalent to hypotenuse(x=point['x'], y=point['y'])
5.0
>>> 
```

---

## Making a Script Executable

`chmod +x`

`$ which python3`

`#!/usr/bin/python3`

`#!/usr/bin/env python3`

---

## Modules

A module is a Python source file that contains only function/class definitions,
not a top-level script.
Modules can be imported by other scripts and modules using the `import` statement.

---

## Modules

For example, the following code can be placed in `vecutils.py`:

```python
def add_vectors(vec1, vec2):
    x1, y1 = vec1
    x2, y2 = vec2
    return x1 + x2, y1 + y2

def mul_vector(vec, m):
    x, y = vec
    return x * m, y * m

def mul_dot(vec1, vec2):
    x1, y1 = vec1
    x2, y2 = vec2
    return (x1 * x2) + (y1 * y2)
```

These functions can then be used from other files or the shell:

```python
>>> import vecutils
>>> vecutils.mul_vector( (3,5), 7 )
(21, 35)
```

---

## Script/Module

A single file can be used both as a module and as a script.
This is done by checking whether the file is being imported.
The special variable `__name__` will be set to the module's name if it were imported,
or the value `__main__` if the module is being run as the top-level script.

Typically, a Python source file will begin with several function definitions,
then will check if it is a script.
The following code performs the check:

```python
if __name__ == '__main__':
    # main code
    pass
```

Some programmers like to define a main function.
This is mostly a matter of personal preference.

```python
def main():
    '''The main function'''

if __name__ == '__main__':
    main()
```

---

## Docstrings, Pydoc, and Doctest

```python
def hypotenuse(x, y):
    """Computes the distance from the origin to the point (x, y)

:param x: the point's x-coordinate
    :param y: the point's y-coordinate
    :return: number. the distance from (0, 0) to the point (x, y)

>>> hypotenuse(3, 4)
    5.0
    >>> hypotenuse(5, 12)
    13.0
    """
    return (x**2 + y**2) ** 0.5

if __name__ == "__main__":
    import doctest
    doctest.testmod(verbose=True)
```

```bash
$ python3 -m pydoc -w hypot
$ python3 -m doctest -v hypot.py
```

---

## Basic Script

```python
#!/usr/bin/env python3

# import statements

# function definitions

if __name__ == '__main__':
    # main/script code
```

---

## Basic Script Using a Separate Main Function

```python
#!/usr/bin/env python3

# import statements

def main():
    # main code

# function definitions

if __name__ == '__main__':
    main()
```

---

## Basic Script Using Doctest

```python
#!/usr/bin/env python3

# import statements

# function definitions

if __name__ == '__main__':
    import doctest
    doctest.testmod()
```

---

# Working With Files

---

Files can be opened with the `open` function.
The contents of the file can be read into a string using `read`.

```python
fl = open('/etc/passwd')
text = fl.read()
fl.close()
```

---

`readlines` returns a list of strings containing every line in the file.

```python
fl = open('/etc/passwd')
lines = fl.readlines()
fl.close()
```

---

Files are iterators over the lines in the file.
This means that file objects can be used with `for` loops.

```python
fl = open('/etc/passwd')
for line in fl:
    print(line)
fl.close()
```

---

## With Statement

The `with` statement ensures that a file is closed when control leaves the block, even on errors.
It is recommended to always use it.

```python
with open('/etc/passwd') as fl:
    for line in fl:
        print(line)
```

---

## Writing Files

The `open` function takes an optional second paramater:
a string containing flags for opening.
The default value is `'r'`, which opens the file for reading.
You can also pass `'w'` to open the file for writing.

```python
with open('output.txt', 'w') as fl:
    print("Hello, world", file=fl)
    fl.write("blah")
```

---

## Binary Files

By default, files are opened in text mode.
In text mode, Python will automatically decode text on read and encode text on write.
Python will also automatically handle conversion to the OS's line ending format.

Files can also be opened in binary mode.
To do this, include a `'b'` in the second argument
(eg `open(filename, 'rb')` or `open(filename, 'wb')`).
With binary files, Python will read and write raw bytes.
The `read` and `write` methods will use the `bytes` type instead of the `str` type.

---

## Unicode

Python's `bytes` type represents a string of bytes.
Python's `str` type represents a string of Unicode code points.
Some (most) code points are too large to represent using one byte,
and there are many ways to encode the code points into multiple bytes.
The `open` function in text mode will attempt to determine the file's encoding and perform the conversion automatically.
However, sometimes the `open` function cannot detect the encoding,
or the byte stream is coming from some other source than a file.
In this case, you must manually encode/decode the `bytes` into a `str`.
`str` has an `encode` method to convert to `bytes`,
and `bytes` has a `decode` method to convert to `str`.

In general, when working with text in Python, you should:

- Decode a byte stream into a `str`
- Work with the text
- Encode the text into `bytes`

---

# Operating System Modules

---

## sys

`sys` contains system-specific info.
Most commonly used contents:

- `sys.argv` is a list containing the command-line parameters.
  You can use it to add command-line options to your program.
  - See also the [argparse](https://docs.python.org/3/library/argparse.html) module.
- `sys.stdin`, `sys.stdout`, and `sys.stderr` are file objects representing the standard streams.
  These can be passed to most functions that expect a file object.
- `sys.path`  is a list of directories where Python will search for modules.

---

## [os.path](https://docs.python.org/3/library/os.path.html)

`os.path` contains functions for manipulating file path names in an OS-independent way.
For example, if you have a folder name in one variable and a file name in another,
you should use `os.path.join` to combine them into a full path name.
`join` will use the correct path separator for the current OS.

`os.path` is actually a wrapper around the `posixpath` and `ntpath` modules.
It imports the correct one for the current OS (`ntpath` for Windows and `posixpath` for MacOS/Linux/other UNIXes).
You can still use those modules directly if you specifically want to use another system's path separator.
In particualr, when working with URLs, you should use `posixpath` to ensure the functions use `'/'` even on Windows.

---

## [os.path](https://docs.python.org/3/library/os.path.html)

```python
>>> dirname = '/home/user'
>>> filename = 'data.txt'
>>> import os.path
>>> os.path.join(dirname, filename)
'/home/user/data.txt'
>>> import ntpath
>>> ntpath.join(r'C:\Users\User', 'data.txt')
'C:\\Users\\User\\data.txt'
>>> base_url = 'http://example.com/archive'
>>> page_name = 'about.html'
>>> import posixpath
>>> posixpath.join(base_url, page_name)
'http://example.com/archive/about.html'
```

---

## [os](https://docs.python.org/3/library/os.html)

`os` contains functions for interacting with the operating system.
The most common uses are file manipulation and environment variables.
Environment variables can be set from a parent process, such as Bash,
and are commonly used on servers for runtime configuration.
The `os` module provides a `dict` called `os.environ` for accessing environment variables.

Useful functions:
- chdir
- listdir
- mkdir/makedirs
- remove/removedirs/rmdir
- rename
- stat
- walk

See also the [shutil](https://docs.python.org/3/library/shutil.html) module,
which provides richer functions to operate on files.

---

## os.stat

`os.stat` returns info about a file or directory.

```python
import os
result = os.stat(os.curdir)
result.st_size # file size
result.st_rsize # "real" file size
result.st_atime # timestamp of most recent access
result.st_mtime # timestamp of most recent modification
```

---

## os.walk

`os.walk` is used to walk a directory tree.
It is used to process every file in every subdirectory of a given root directory.

```python
import os
root_path = '/home/user'
for current_dir, subdirs, files in os.walk(root_path):
    for filename in files:
        file_path = os.path.join(current_dir, filename)
        # ...
```

---

## [cmd](https://docs.python.org/3/library/cmd.html)

The Cmd class provides a simple framework for writing line-oriented command interpreters. These are often useful for test harnesses, administrative tools, and prototypes that will later be wrapped in a more sophisticated interface.

---

## [subprocess](https://docs.python.org/3/library/subprocess.html)

The `subprocess` module enables you to run external programs from Python.
You can set environment variables for the programs, pass command-line arguments to them, and connect to their standard input, output, and error streams.

```python
import subprocess
subprocess.run(["ls", "-l"])
proc = subprocess.Popen(['ssh'],
  stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
sout, serr = proc.communicate('command')
```

---

# Text Modules

---

## [re](https://docs.python.org/3/library/re.html)

`re` is Python's regular expression processor.
The `match` and `search` functions work similarly to Grep.
They return a match object if there was a match, or None if there was no match.
The difference is `match` tries to match the pattern at the beginning of the string,
while `search` tries to match the pattern anywhere in the string.
There is also a `sub` function which works like Sed.

```python
with open('/usr/share/dict/words') as f:
    for line in f:
        m = re.match(r'\bc...h\b', line.strip())
        if m:
            print(m.string)

with open('/etc/passwd') as f:
  for line in f:
      line = re.sub(r'([^:]*):.*:([^:]*)', r'\1 uses \2', line.strip())
      print(line)

```

---

## Compiled Regular Expressions

```python
ch = re.compile(r'\bc...h\b')
sh = re.compile('([^:]*):.*:([^:])')

with open('/usr/share/dict/words') as f:
    for line in f:
        m = ch.match(line.strip())
        if m:
            print(m.string)

with open('/etc/passwd') as f:
  for line in f:
      line = sh.sub(r'\1 uses \2', line.strip())
      print(line)
```

---

# Utility Modules

---

## [math](https://docs.python.org/3/library/math.html)

`math` contains several mathematical functions.
Notably, there are `hypot`, `degrees`, `radians` functions,
which other languages often don't have.

---

## [decimal](https://docs.python.org/3/library/decimal.html)

`decimal` implements an arbitrary-precision decimal datatype.
Unlike `float`, `decimal` does not produce rounding errors.

---

## [random](https://docs.python.org/3/library/random.html)

`random` implements a random-number generator and several helper functions for it.

```python
>>> import random
>>> random.random()
0.8213459268075176
>>> random.randint(1, 10)
9
>>> random.randrange(5)
4
>>> nums = [1, 2, 3, 4, 5]
>>> random.choice(nums)
3
>>> random.shuffle(nums)
>>> nums
[5, 3, 1, 2, 4]
```

---

## [pprint](https://docs.python.org/3/library/pprint.html)

`pprint` contains a pretty-printer function.
It will word-wrap long lists and sort dictionary keys.

```python
>>> from pprint import pprint
>>> lst = list(range(50))
>>> print(lst)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
>>> pprint(lst)
[0,
 1,
 2,
 3,
 4,
 5,
 6,
 7,
 8,
 9,
 10,
 11,
 12,
 13,
 14,
 15,
 16,
 17,
 18,
 19,
 20,
 21,
 22,
 23,
 24,
 25,
 26,
 27,
 28,
 29,
 30,
 31,
 32,
 33,
 34,
 35,
 36,
 37,
 38,
 39,
 40,
 41,
 42,
 43,
 44,
 45,
 46,
 47,
 48,
 49]
>>> 
```

---

# File Format Modules

---

## [csv](https://docs.python.org/3/library/csv.html)

`csv` contains datatypes for reading and writing CSV files.
The basic `reader` and `writer` types use lists.
The `DictReader` and `DictWriter` types use dictionaries.

```python
import csv

with open('MOCK_DATA.csv') as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row['last_name'])

```

---

## [configparser](https://docs.python.org/3/library/configparser.html)

This module provides the ConfigParser class which implements a basic configuration language which provides a structure similar to what’s found in Microsoft Windows INI files. You can use this to write Python programs which can be customized by end users easily.

---

## [json](https://docs.python.org/3/library/json.html)

JSON is a text-based data interchange format based on JavaScript.
JSON supports booleans, numbers, strings, arrays, and objects (key-value pairs).
The `json` module provides a simple mapping between JSON types and Python built-in types.

```python
>>> import json
>>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
'["foo", {"bar": ["baz", null, 1.0, 2]}]'
>>> print(json.dumps("\"foo\bar"))
"\"foo\bar"
>>> print(json.dumps('\u1234'))
"\u1234"
>>> print(json.dumps('\\'))
"\\"
>>> print(json.dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True))
{"a": 0, "b": 0, "c": 0}
```

---

## [sqlite3](https://docs.python.org/3/library/sqlite3.html)

SQLite is a C library that provides a lightweight disk-based database that doesn’t require a separate server process and allows accessing the database using a nonstandard variant of the SQL query language. Some applications can use SQLite for internal data storage. It’s also possible to prototype an application using SQLite and then port the code to a larger database such as PostgreSQL or Oracle.

---

# Networking Modules

---

## [urllib](https://docs.python.org/3/library/urllib.html)

```python
>>> from urllib.request import urlopen
>>> response = urlopen('http://python.org')
>>> response.status
200
>>> data = response.read()
```

See also [requests](http://docs.python-requests.org/en/master/)

---

Others

- `datetime`: date and time functions
- `collections`: specialized collection types
- `fileinput`: implements a helper class and functions to quickly write a loop over standard input or a list of files
- `glob`: functions to expand glob patterns like Bash
- `pickle`/`shelve`: save and read Python objects
- `zipfile`/`tarfile`: working with compressed archive files
- `logging`: a logging system for server/background/long-running programs
- `curses`: interface for advanced terminal interaction
- `smtp`: write and send emails
- `ctypes`: call functions defined in C DLLs

---

# Virtual Environments

---

```bash
$ python -m venv myenv
$ source myenv/bin/activate
```

```dos
> python -m venv myenv
> myenv\Scripts\activate.bat
```

---

# Pip

---

# Popular Third-Party Modules

---

## [Flask](http://flask.pocoo.org/)

```python
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"
```

```bash
$ pip install flask
$ FLASK_APP=hello.py flask run
```

---

## [SQLAlchemy](http://www.sqlalchemy.org/)

---

## [Pillow](http://pillow.readthedocs.io/en/latest/)

```python
from math import pi, sin, cos
from PIL import Image

def plotIntensity(exp, pixelsPerUnit = 150):
    canvasWidth = 2 * pixelsPerUnit + 1
    canvas = Image.new("L", (canvasWidth, canvasWidth))

for py in range(canvasWidth):
        for px in range(canvasWidth):
            # Convert pixel location to [-1,1] coordinates
            x = float(px - pixelsPerUnit) / pixelsPerUnit 
            y = -float(py - pixelsPerUnit) / pixelsPerUnit
            z = exp(x,y)
            # Scale [-1,1] result to [0,255].
            intensity = int(z * 127.5 + 127.5)
            canvas.putpixel((px,py), intensity)
    return canvas

def plotColor(pixelsPerUnit = 150):
    redPlane   = plotIntensity(lambda x, y: sin(pi*x), pixelsPerUnit)
    greenPlane = plotIntensity(lambda x, y: cos(pi*x*y), pixelsPerUnit)
    bluePlane  = plotIntensity(lambda x, y: sin(pi*y), pixelsPerUnit)
    return Image.merge("RGB", (redPlane, greenPlane, bluePlane))

if __name__ == '__main__':
    image = plotColor()
    image.save("img.png", "PNG")

```

---

## [bs4](https://www.crummy.com/software/BeautifulSoup/)

```python
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
The Dormouse's story

Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.

...
"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')
soup.find_all('a')

```

---

## [matplotlib](https://matplotlib.org/)

```python
import matplotlib.pyplot as plt
xs = [1, 2, 3, 4]
ys = [1, 4, 9, 16]
plt.plot(xs, ys)
plt.ylabel('some numbers')
plt.show()
```

---

## [requests](http://docs.python-requests.org/en/master/)

```python
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}
```

---

# Other Resources

- http://saddlebackcss.github.io/tutorials/python/basic/2016/01/07/python-basics-0
- https://anandology.com/python-practice-book/index.html
- https://learnpythonthehardway.org/python3/
- https://www.linuxjournal.com/content/python-scripts-replacement-bash-utility-scripts
- [Professional System Administration with Python](http://sysadminpy.com/)
- OReilly - Python for Unix and Linux System Administration