Text

Objectives

• [ ] String indexing and immutability

• [ ] The string module

• [ ] Manipulating text strings

• [ ] Formatting text strings

• [ ] Text I/O

• [ ] Working with Unicode

• [ ] Text to arrays

Strings

• Point index

• Interval index

• Negative index

• Stride

• Reversing a string

• Strings are immutable

References

• String formats

• String methods

• Regular expressions

• File I/O

s = "hello world"

s[0], s[6]

('h', 'w')

s[0:6]

'hello '

s[-1], s[-3]

('d', 'r')

s[::2]

'hlowrd'

s[::-1]

'dlrow olleh'

try:
    s[0] = 'H'
except TypeError as e:
    print(e)

'str' object does not support item assignment

The string module

• String constants

• String capwords

import string

string.digits

'0123456789'

string.ascii_letters

'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'

string.punctuation

'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

string.whitespace

' \t\n\r\x0b\x0c'

string.printable

'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c'

Orphan function in strings module

string.capwords(s)

'Hello World'

String methods

Methods to change case

s.upper()

'HELLO WORLD'

s.lower()

'hello world'

'ß'.casefold()

'ss'

s.capitalize()

'Hello world'

s.title()

'Hello World'

Difference between title method and capwords function

'hello:world'.title()

'Hello:World'

string.capwords('hello:world')

'Hello:world'

string.capwords('hello:world', sep=':')

'Hello:World'

String predicates

s.isalnum()

False

s.isalpha()

False

s.isnumeric()

False

s.isidentifier()

False

s.isprintable()

True

s.startswith('hell')

True

s.endswith('ld')

True

Searching and counting

'llo' in s

True

'foo' in s

False

s.find('llo')

2

s.index('llo')

2

s.find('foo')

-1

try:
    s.index('foo')
except ValueError as e:
    print(e)

substring not found

s.count('l')

3

s.count('ll')

1

Stripping

'   hello world   '.strip()

'hello world'

'   hello world   '.lstrip()

'hello world   '

'   hello world   '.rstrip()

'   hello world'

Splitting and joining

s.split()

['hello', 'world']

s.split('l')

['he', '', 'o wor', 'd']

'-'.join(s)

'h-e-l-l-o- -w-o-r-l-d'

'-'.join(s.split())

'hello-world'

'l'.join(s.split('l'))

'hello world'

Translation

'GATTACA'.translate(str.maketrans('ACTG', 'TAGC'))

'CTGGTAT'

'GATTACA'.translate(str.maketrans('', '', 'AC'))

'GTT'

'GATTACA'.translate(str.maketrans(string.ascii_uppercase, string.ascii_lowercase))

'gattaca'

ord and chr

ord('A'), ord('a')

(65, 97)

chr(65), chr(97)

('A', 'a')

chr(ord('B') + (ord('a') - ord('A')))

'b'

Formatting strings

C sytle formatting

pi = 3.141592653589793
r = 2

'area = %f * %d^2' % (pi, r)

'area = 3.141593 * 2^2'

Precision and padding

'area = %8.2f * %03d^2' % (pi, r)

'area =     3.14 * 002^2'

Right align string

'%10s = %8.2f * %03d^2' % ('area', pi, r)

'      area =     3.14 * 002^2'

Left align string

'%-10s = %8.2f * %03d^2' % ('area', pi, r)

'area       =     3.14 * 002^2'

Using the format method

"{" [field_name] ["!" conversion] [":" format_spec] "}"

'{big:,}'.format(big=int(1e9))

'1,000,000,000'

'{pct:.1%}'.format(pct=0.5)

'50.0%'

'area = {} * {}^2'.format(pi, r)

'area = 3.141592653589793 * 2^2'

'area = {a} * {b}^2'.format(a=pi, b=r)

'area = 3.141592653589793 * 2^2'

'area = {pi:8,.4} * {r:06d}^2'.format(pi=pi, r=r)

'area =    3.142 * 000002^2'

'{:>10}'.format('area')

'      area'

'{:<10}'.format('area')

'area      '

'{:^10}'.format('area')

'   area   '

'{:=^10}'.format('area')

'===area==='

import datetime

now = datetime.datetime.now()
'{:%a, %d %b %Y: %H:%M %p}'.format(now)

'Mon, 13 Jan 2020: 10:41 AM'

Using f strings

f'area = {pi} * {r}^2'

'area = 3.141592653589793 * 2^2'

x = 'area'
f'{x:=^10}'

'===area==='

Templates

from string import Template

t = Template("$who likes $what")
items = [('ann', 'Python'), ('bob', 'R'), ('cody', 'C++')]
for name, lang in items:
    print(t.substitute(who=name, what=lang))

ann likes Python
bob likes R
cody likes C++

items = [('ann', 'Python'), ('bob', 'R'), ('cody', 'C++')]
for name, lang in items:
    print("{} likes {}".format(name, lang))

ann likes Python
bob likes R
cody likes C++

items = [('ann', 'Python'), ('bob', 'R'), ('cody', 'C++')]
for name, lang in items:
    print(f"{name} likes {lang}")

ann likes Python
bob likes R
cody likes C++

Encodings

1. There ain’t no such thing as plain text

2. Text is composed of

   1. Letters (Platonic ideal)

   2. Code points (an integer)

   3. Encodings (how the integer is written in memory)

3. Python 3 defaults to Unicode UTF-8 encoding (Unicode code points in Python look like \uxxxx where x is hexadecimal)

4. To see the bytes from an encoding, use the encode method

5. To see the letter from bytes, use the decode method

6. You can specify the encoding as an optional argument in the open function

7. You can use Unicode in variable names

Unicode strings

print('hello \u732b')

hello 猫

s = '猫'
print(f'hello {s}')

hello 猫

Byte strings

kitty = '小' + '猫'

print(f'hello {kitty}')

hello 小猫

kitty_bytes = kitty.encode('utf8')
kitty_bytes

b'\xe5\xb0\x8f\xe7\x8c\xab'

kitty_bytes.decode('utf8')

'小猫'

try:
    kitty_bytes.decode('ascii')
except UnicodeDecodeError as e:
    print(e)

'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range(128)

for suit in '\u2660 \u2665  \u2666  \u2663  \u2664  \u2661  \u2662  \u2667'.split():
    print(suit, end=',')

♠,♥,♦,♣,♤,♡,♢,♧,

Unicode variable names

αβγδϵ = 23

ΑΒΓΔΕ = 42

(ŷ, ÿ, ỹ, ȳ, y̅, y̆, y̌, y̲) = range(8)

α⃗, α⃖, α⃡, α⃐, α⃑ = range(5)

ℜ = 'real'

Reading and writing text files

%%file haiku.txt
古池や蛙飛び込む水の音
ふるいけやかわずとびこむみずのおと

Overwriting haiku.txt

Context manager

A context manager is a class that does some automatic activity on entry and exit. It can be written most easily using a standard library decorator.

class CMDemo:
    """Demo of context manager."""

    def __init__(self, name):
        self.name = name

    def __enter__(self):
        print("Entering %s" % self.name)

    def __exit__(self, *args):
        print("Exiting %s" % self.name)

with CMDemo('foo'):
    print('foo')

Entering foo
foo
Exiting foo

from contextlib import contextmanager

@contextmanager
def tag(name):
    print("<%s>" % name)
    yield
    print("</%s>" % name)

with tag('foo'):
    print('Hello')

<foo>
Hello
</foo>

It is good practice to use open as a context manager for file I/O so we don’t forget to close it. The old practice looks something like

f = open('foo.txt')
# do stuff with f over many lines
f.close()

The trouble is that people forget to close f and there are only a finite number of file handlers provided by the operating system, and programs can crash if that number is exceeded.

with open('haiku.txt') as f:
    for line in f:
        print(line, end='')

古池や蛙飛び込む水の音
ふるいけやかわずとびこむみずのおと

with open('haiku.txt') as f:
    haiku = f.read()

haiku

'古池や蛙飛び込む水の音\nふるいけやかわずとびこむみずのおと\n'

haiku.split()

['古池や蛙飛び込む水の音', 'ふるいけやかわずとびこむみずのおと']

with open('haiku_alt.txt', 'w') as f:
    f.write(haiku)

! cat haiku_alt.txt

古池や蛙飛び込む水の音
ふるいけやかわずとびこむみずのおと

Using regular expressions

• Read Regular Expression HOWTO

• Practice at https://regex101.com

• Play RegEx Golf

import re

Matching Characters

beer = '''99 bottles of Beer on the wall, 99 bottles of beeR.
Take one down and pass it around, 98 bottles of beer on the wall.'''

re.findall('beer', beer)

['beer']

re.findall('beer', beer, re.IGNORECASE)

['Beer', 'beeR', 'beer']

re.findall('on', beer)

['on', 'on', 'on']

Alternatives

re.findall('bottles|beer', beer, re.IGNORECASE)

['bottles', 'Beer', 'bottles', 'beeR', 'bottles', 'beer']

Word boundaries

re.findall(r'\bon\b', beer)

['on', 'on']

re.findall(r'.', beer)[-10:]

[' ', 't', 'h', 'e', ' ', 'w', 'a', 'l', 'l', '.']

Character sets

re.findall(r'\d', beer)

['9', '9', '9', '9', '9', '8']

re.findall(r'[0-9]', beer)

['9', '9', '9', '9', '9', '8']

re.findall(r'\w', beer)[11:25]

['B', 'e', 'e', 'r', 'o', 'n', 't', 'h', 'e', 'w', 'a', 'l', 'l', '9']

Repeating Things

re.findall(r'\d+', beer)

['99', '99', '98']

re.findall(r'b.+r', beer)

['bottles of Beer', 'bottles of beer']

re.findall(r'be+', beer)

['bee', 'bee']

re.findall(r'be*', beer)

['b', 'b', 'bee', 'b', 'bee']

re.findall(r'b[aeiou]+', beer)

['bo', 'bo', 'bee', 'bo', 'bee']

re.findall(r'b[aeiou]{2,}', beer)

['bee', 'bee']

re.findall(r'b[aeiou]{1}', beer)

['bo', 'bo', 'be', 'bo', 'be']

Finding matches

for m in re.finditer('beer', beer, re.IGNORECASE):
    print(m.start(), m.end(), m.span(),  m.group())

14 18 (14, 18) Beer
46 50 (46, 50) beeR
100 104 (100, 104) beer

Grouping

re.findall(r'(\d+)\s+(\b\w+?\b)', beer, re.IGNORECASE)

[('99', 'bottles'), ('99', 'bottles'), ('98', 'bottles')]

Splitting

re.split(r'\d+', beer)

['',
 ' bottles of Beer on the wall, ',
 ' bottles of beeR.\nTake one down and pass it around, ',
 ' bottles of beer on the wall.']

Search and replace

print(re.sub('beer', 'whiskey', beer, flags=re.IGNORECASE))

99 bottles of whiskey on the wall, 99 bottles of whiskey.
Take one down and pass it around, 98 bottles of whiskey on the wall.

print(re.sub(r'(\d+)\s+(\b\w+?\b)', r'\2 \1', beer, re.IGNORECASE))

bottles 99 of Beer on the wall, bottles 99 of beeR.
Take one down and pass it around, 98 bottles of beer on the wall.

Function versus compiled method

pattern = re.compile(r'(\d+)\s+(\b\w+?\b)')
pattern.findall(beer)

[('99', 'bottles'), ('99', 'bottles'), ('98', 'bottles')]

Raw strings

The backslash \ is an escape character in a regular Python string. So we need to escape it to match a literal \. However, \ is an escape character in the regular expression mini-language when compiling the regular expression pattern. So we need to escape at two levels - hence we need \\\\ to match a literal \. The raw string rfoo treats \ as a literal character rather than an escape character.

latex = 'latex uses \section over and over again like so \section'

re.findall('\section', latex)

[]

re.findall('\\section', latex)

[]

re.findall('\\\\section', latex)

['\\section', '\\section']

re.findall(r'\\section', latex)

['\\section', '\\section']

Examples

Removing punctuation

ss = 'What the #$@&%*! does your code mean?'

Using a comprehension

''.join(s for s in ss if not s in string.punctuation)

'What the  does your code mean'

Using a built-in function

ss.translate(str.maketrans('','', string.punctuation))

'What the  does your code mean'

Using a regular expression

pat = re.compile('[%s]' % re.escape(string.punctuation))

pat.sub('', ss)

'What the  does your code mean'

Timing

%timeit ''.join(s for s in ss if not s in string.punctuation)

6.65 µs ± 25.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit ss.translate(str.maketrans('','', string.punctuation))

4.25 µs ± 32.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit pat.sub('', ss)

2.17 µs ± 25.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Custom version of capwords

string.capwords('hello    world')

'Hello World'

def my_capwords(ss):
    return ' '.join([s.title() for s in ss.split()])

my_capwords('hello    world')

'Hello World'

Bag of words

Create a table of counts, where rows represent unique words and columns represent different documents. Ignore case and capitalization.

doc1 = """The wheels on the bus go,
Round and round,
Round and round,
Round and round.
The wheels on the bus go
Round and round,
All through the town."""

doc2 = """The doors on the bus go,
Open and shut,
Open and shut,
Open and shut.
The doors on the bus go
Open and shut,
All through the town."""

doc3 = """The Driver on the bus says,
"Move on back!
Move on back!
Move on back!"
The Driver on the bus says,
"Move on back!"
All through the town."""

doc4 = """The babies on the bus go,
"Wah, wah, wah!
Wah, wah, wah!
Wah, wah, wah!"
The babies on the bus go,
"Wah, wah, wah!"
All through the town."""

docs = [doc1, doc2, doc3, doc4]
doc_words = [doc.strip().lower().translate(str.maketrans('', '', string.punctuation)).split()
             for doc in docs]
words = [word for words in doc_words for word in words]
vocab = set(words)

import numpy as np
import pandas as pd

table = np.zeros((len(vocab), len(docs)), dtype='int')

for i, word in enumerate(vocab):
    for j, doc in enumerate(doc_words):
        table[i, j] = doc.count(word)

pd.DataFrame(table, columns='doc1 doc2 doc3 doc4'.split(), index=vocab)

	doc1	doc2	doc3	doc4
town	1	1	1	1
round	8	0	0	0
doors	0	2	0	0
says	0	0	2	0
on	2	2	6	2
and	4	4	0	0
through	1	1	1	1
open	0	4	0	0
move	0	0	4	0
back	0	0	4	0
babies	0	0	0	2
wah	0	0	0	12
bus	2	2	2	2
the	5	5	5	5
driver	0	0	2	0
shut	0	4	0	0
all	1	1	1	1
wheels	2	0	0	0
go	2	2	0	2