Strings¶
The process of cleaning data for analysis often requires working with text, for example, to correct typos, convert to standard nomenclature and resolve ambiguous labels. In some statistical fields that deal with (say) processing electronic medical records, information science or recommendations based on user feedback, text must be processed before analysis - for example, by converting to a bag of words.
We will use a whimsical example to illustrate Python tools for munging text data using string methods and regular expressions. Finally, we will see how to format text data for reporting.
Get “Through the Looking Glass”¶
import requests
try:
with open('looking_glass.txt') as f:
text = f.read()
except IOError:
url = 'http://www.gutenberg.org/cache/epub/12/pg12.txt'
res = requests.get(url)
text = res.text
with open('looking_glass.txt', 'w') as f:
f.write(str(text))
Slice to get Jabberwocky¶
start = text.find('JABBERWOCKY')
text[start:start+2000]
"JABBERWOCKYrnrn 'Twas brillig, and the slithy tovesrn Did gyre and gimble in the wabe;rn All mimsy were the borogoves,rn And the mome raths outgrabe.rnrn 'Beware the Jabberwock, my son!rn The jaws that bite, the claws that catch!rn Beware the Jubjub bird, and shunrn The frumious Bandersnatch!'rnrn He took his vorpal sword in hand:rn Long time the manxome foe he sought--rn So rested he by the Tumtum tree,rn And stood awhile in thought.rnrn And as in uffish thought he stood,rn The Jabberwock, with eyes of flame,rn Came whiffling through the tulgey wood,rn And burbled as it came!rnrn One, two! One, two! And through and throughrn The vorpal blade went snicker-snack!rn He left it dead, and with its headrn He went galumphing back.rnrn 'And hast thou slain the Jabberwock?rn Come to my arms, my beamish boy!rn O frabjous day! Callooh! Callay!'rn He chortled in his joy.rnrn 'Twas brillig, and the slithy tovesrn Did gyre and gimble in the wabe;rn All mimsy were the borogoves,rn And the mome raths outgrabe.rnrnrn'It seems very pretty,' she said when she had finished it, 'but it'srnRATHER hard to understand!' (You see she didn't like to confess, evenrnto herself, that she couldn't make it out at all.) 'Somehow it seemsrnto fill my head with ideas--only I don't exactly know what they are!rnHowever, SOMEBODY killed SOMETHING: that's clear, at any rate--'rnrn'But oh!' thought Alice, suddenly jumping up, 'if I don't make haste Irnshall have to go back through the Looking-glass, before I've seen whatrnthe rest of the house is like! Let's have a look at the garden first!'rnShe was out of the room in a moment, and ran down stairs--or, at least,rnit wasn't exactly running, but a new invention of hers for getting downrnstairs quickly and easily, as Alice said to herself. She just kept therntips of her fingers on the hand-rail, and floated gently down withoutrneven"
end = text.find('It seems very pretty', start)
poem = text[start:end]
poem
"JABBERWOCKYrnrn 'Twas brillig, and the slithy tovesrn Did gyre and gimble in the wabe;rn All mimsy were the borogoves,rn And the mome raths outgrabe.rnrn 'Beware the Jabberwock, my son!rn The jaws that bite, the claws that catch!rn Beware the Jubjub bird, and shunrn The frumious Bandersnatch!'rnrn He took his vorpal sword in hand:rn Long time the manxome foe he sought--rn So rested he by the Tumtum tree,rn And stood awhile in thought.rnrn And as in uffish thought he stood,rn The Jabberwock, with eyes of flame,rn Came whiffling through the tulgey wood,rn And burbled as it came!rnrn One, two! One, two! And through and throughrn The vorpal blade went snicker-snack!rn He left it dead, and with its headrn He went galumphing back.rnrn 'And hast thou slain the Jabberwock?rn Come to my arms, my beamish boy!rn O frabjous day! Callooh! Callay!'rn He chortled in his joy.rnrn 'Twas brillig, and the slithy tovesrn Did gyre and gimble in the wabe;rn All mimsy were the borogoves,rn And the mome raths outgrabe.rnrnrn'"
print(poem)
JABBERWOCKY
'Twas brillig, and the slithy toves
Did gyre and gimble in the wabe;
All mimsy were the borogoves,
And the mome raths outgrabe.
'Beware the Jabberwock, my son!
The jaws that bite, the claws that catch!
Beware the Jubjub bird, and shun
The frumious Bandersnatch!'
He took his vorpal sword in hand:
Long time the manxome foe he sought--
So rested he by the Tumtum tree,
And stood awhile in thought.
And as in uffish thought he stood,
The Jabberwock, with eyes of flame,
Came whiffling through the tulgey wood,
And burbled as it came!
One, two! One, two! And through and through
The vorpal blade went snicker-snack!
He left it dead, and with its head
He went galumphing back.
'And hast thou slain the Jabberwock?
Come to my arms, my beamish boy!
O frabjous day! Callooh! Callay!'
He chortled in his joy.
'Twas brillig, and the slithy toves
Did gyre and gimble in the wabe;
All mimsy were the borogoves,
And the mome raths outgrabe.
'
print(poem.title())
Jabberwocky
'Twas Brillig, And The Slithy Toves
Did Gyre And Gimble In The Wabe;
All Mimsy Were The Borogoves,
And The Mome Raths Outgrabe.
'Beware The Jabberwock, My Son!
The Jaws That Bite, The Claws That Catch!
Beware The Jubjub Bird, And Shun
The Frumious Bandersnatch!'
He Took His Vorpal Sword In Hand:
Long Time The Manxome Foe He Sought--
So Rested He By The Tumtum Tree,
And Stood Awhile In Thought.
And As In Uffish Thought He Stood,
The Jabberwock, With Eyes Of Flame,
Came Whiffling Through The Tulgey Wood,
And Burbled As It Came!
One, Two! One, Two! And Through And Through
The Vorpal Blade Went Snicker-Snack!
He Left It Dead, And With Its Head
He Went Galumphing Back.
'And Hast Thou Slain The Jabberwock?
Come To My Arms, My Beamish Boy!
O Frabjous Day! Callooh! Callay!'
He Chortled In His Joy.
'Twas Brillig, And The Slithy Toves
Did Gyre And Gimble In The Wabe;
All Mimsy Were The Borogoves,
And The Mome Raths Outgrabe.
'
poem.count('the')
15
print(poem.replace('the', 'XXX'))
JABBERWOCKY
'Twas brillig, and XXX slithy toves
Did gyre and gimble in XXX wabe;
All mimsy were XXX borogoves,
And XXX mome raths outgrabe.
'Beware XXX Jabberwock, my son!
The jaws that bite, XXX claws that catch!
Beware XXX Jubjub bird, and shun
The frumious Bandersnatch!'
He took his vorpal sword in hand:
Long time XXX manxome foe he sought--
So rested he by XXX Tumtum tree,
And stood awhile in thought.
And as in uffish thought he stood,
The Jabberwock, with eyes of flame,
Came whiffling through XXX tulgey wood,
And burbled as it came!
One, two! One, two! And through and through
The vorpal blade went snicker-snack!
He left it dead, and with its head
He went galumphing back.
'And hast thou slain XXX Jabberwock?
Come to my arms, my beamish boy!
O frabjous day! Callooh! Callay!'
He chortled in his joy.
'Twas brillig, and XXX slithy toves
Did gyre and gimble in XXX wabe;
All mimsy were XXX borogoves,
And XXX mome raths outgrabe.
'
Find palindromic words in poem if any¶
poem = poem.lower()
import string
string.punctuation
'!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~'
poem = poem.translate(dict.fromkeys(map(ord, string.punctuation)))
poem
'jabberwockyrnrn twas brillig and the slithy tovesrn did gyre and gimble in the wabern all mimsy were the borogovesrn and the mome raths outgrabernrn beware the jabberwock my sonrn the jaws that bite the claws that catchrn beware the jubjub bird and shunrn the frumious bandersnatchrnrn he took his vorpal sword in handrn long time the manxome foe he soughtrn so rested he by the tumtum treern and stood awhile in thoughtrnrn and as in uffish thought he stoodrn the jabberwock with eyes of flamern came whiffling through the tulgey woodrn and burbled as it camernrn one two one two and through and throughrn the vorpal blade went snickersnackrn he left it dead and with its headrn he went galumphing backrnrn and hast thou slain the jabberwockrn come to my arms my beamish boyrn o frabjous day callooh callayrn he chortled in his joyrnrn twas brillig and the slithy tovesrn did gyre and gimble in the wabern all mimsy were the borogovesrn and the mome raths outgrabernrnrn'
words = poem.split()
words[:10]
['jabberwocky',
'twas',
'brillig',
'and',
'the',
'slithy',
'toves',
'did',
'gyre',
'and']
def is_palindrome(word):
return word == word[::-1]
{word for word in words if is_palindrome(word)}
{'did', 'o'}
Top 10 most frequent words¶
import collections
poem_counter = collections.Counter(words)
poem_counter.most_common(10)
[('the', 19),
('and', 14),
('he', 7),
('in', 6),
('my', 3),
('through', 3),
('jabberwock', 3),
('gyre', 2),
('mimsy', 2),
('mome', 2)]
Words that appear exactly twice.¶
[(k, v) for (k, v) in poem_counter.items() if v==2]
[('gyre', 2),
('mimsy', 2),
('mome', 2),
('wabe', 2),
('gimble', 2),
('it', 2),
('outgrabe', 2),
('all', 2),
('came', 2),
('with', 2),
('thought', 2),
('twas', 2),
('that', 2),
('vorpal', 2),
('were', 2),
('brillig', 2),
('slithy', 2),
('stood', 2),
('as', 2),
('did', 2),
('toves', 2),
('borogoves', 2),
('two', 2),
('beware', 2),
('went', 2),
('raths', 2),
('his', 2),
('one', 2)]
Trigrams¶
All possible sequences of 3 words in the poem.
list(zip(words[:], words[1:], words[2:]))[:10]
[('jabberwocky', 'twas', 'brillig'),
('twas', 'brillig', 'and'),
('brillig', 'and', 'the'),
('and', 'the', 'slithy'),
('the', 'slithy', 'toves'),
('slithy', 'toves', 'did'),
('toves', 'did', 'gyre'),
('did', 'gyre', 'and'),
('gyre', 'and', 'gimble'),
('and', 'gimble', 'in')]
import itertools as it
def window(x, n):
"""Sliding widnow of size n from iterable x."""
s = (it.islice(x, i, None) for i in range(n))
return zip(*s)
list(window(words, 3))[:10]
[('jabberwocky', 'twas', 'brillig'),
('twas', 'brillig', 'and'),
('brillig', 'and', 'the'),
('and', 'the', 'slithy'),
('the', 'slithy', 'toves'),
('slithy', 'toves', 'did'),
('toves', 'did', 'gyre'),
('did', 'gyre', 'and'),
('gyre', 'and', 'gimble'),
('and', 'gimble', 'in')]
Find words in poem that are over-represented¶
book = text
book = book.lower().translate(dict.fromkeys(map(ord, string.punctuation)))
book_counter = collections.Counter(book.split())
n = sum(book_counter.values())
book_freqs = {k: v/n for k, v in book_counter.items()}
n = sum(poem_counter.values())
stats = [(k, v, book_freqs.get(k,0)*n) for k, v in poem_counter.items()]
from pandas import DataFrame
df = DataFrame(stats, columns = ['word', 'observed', 'expected'])
df['score'] = (df.observed-df.expected)**2/df.expected
df = df.sort_values(['score'], ascending=False)
df.head(n=10)
| word | observed | expected | score | |
|---|---|---|---|---|
| 55 | jabberwock | 3 | 0.01557 | 572.045510 |
| 46 | vorpal | 2 | 0.01038 | 381.363673 |
| 69 | beware | 2 | 0.01038 | 381.363673 |
| 66 | borogoves | 2 | 0.01557 | 252.917766 |
| 43 | twas | 2 | 0.01557 | 252.917766 |
| 65 | whiffling | 1 | 0.00519 | 190.681837 |
| 39 | tumtum | 1 | 0.00519 | 190.681837 |
| 42 | burbled | 1 | 0.00519 | 190.681837 |
| 52 | snickersnack | 1 | 0.00519 | 190.681837 |
| 54 | callooh | 1 | 0.00519 | 190.681837 |
Encode and decode poem using a Caesar cipher¶
print(poem)
jabberwocky
twas brillig and the slithy toves
did gyre and gimble in the wabe
all mimsy were the borogoves
and the mome raths outgrabe
beware the jabberwock my son
the jaws that bite the claws that catch
beware the jubjub bird and shun
the frumious bandersnatch
he took his vorpal sword in hand
long time the manxome foe he sought
so rested he by the tumtum tree
and stood awhile in thought
and as in uffish thought he stood
the jabberwock with eyes of flame
came whiffling through the tulgey wood
and burbled as it came
one two one two and through and through
the vorpal blade went snickersnack
he left it dead and with its head
he went galumphing back
and hast thou slain the jabberwock
come to my arms my beamish boy
o frabjous day callooh callay
he chortled in his joy
twas brillig and the slithy toves
did gyre and gimble in the wabe
all mimsy were the borogoves
and the mome raths outgrabe
Encoding¶
def encode(text, k):
table = dict(zip(map(ord, string.ascii_lowercase),
string.ascii_lowercase[k:] + string.ascii_lowercase[:k]))
return text.translate(table)
cipher = encode(poem, 2)
print(cipher)
lcddgtyqema
vycu dtknnki cpf vjg unkvja vqxgu
fkf iatg cpf ikodng kp vjg ycdg
cnn okoua ygtg vjg dqtqiqxgu
cpf vjg oqog tcvju qwvitcdg
dgyctg vjg lcddgtyqem oa uqp
vjg lcyu vjcv dkvg vjg encyu vjcv ecvej
dgyctg vjg lwdlwd dktf cpf ujwp
vjg htwokqwu dcpfgtupcvej
jg vqqm jku xqtrcn uyqtf kp jcpf
nqpi vkog vjg ocpzqog hqg jg uqwijv
uq tguvgf jg da vjg vwovwo vtgg
cpf uvqqf cyjkng kp vjqwijv
cpf cu kp whhkuj vjqwijv jg uvqqf
vjg lcddgtyqem ykvj gagu qh hncog
ecog yjkhhnkpi vjtqwij vjg vwniga yqqf
cpf dwtdngf cu kv ecog
qpg vyq qpg vyq cpf vjtqwij cpf vjtqwij
vjg xqtrcn dncfg ygpv upkemgtupcem
jg nghv kv fgcf cpf ykvj kvu jgcf
jg ygpv icnworjkpi dcem
cpf jcuv vjqw unckp vjg lcddgtyqem
eqog vq oa ctou oa dgcokuj dqa
q htcdlqwu fca ecnnqqj ecnnca
jg ejqtvngf kp jku lqa
vycu dtknnki cpf vjg unkvja vqxgu
fkf iatg cpf ikodng kp vjg ycdg
cnn okoua ygtg vjg dqtqiqxgu
cpf vjg oqog tcvju qwvitcdg
Decoding¶
recovered = encode(cipher, -2)
print(recovered)
jabberwocky
twas brillig and the slithy toves
did gyre and gimble in the wabe
all mimsy were the borogoves
and the mome raths outgrabe
beware the jabberwock my son
the jaws that bite the claws that catch
beware the jubjub bird and shun
the frumious bandersnatch
he took his vorpal sword in hand
long time the manxome foe he sought
so rested he by the tumtum tree
and stood awhile in thought
and as in uffish thought he stood
the jabberwock with eyes of flame
came whiffling through the tulgey wood
and burbled as it came
one two one two and through and through
the vorpal blade went snickersnack
he left it dead and with its head
he went galumphing back
and hast thou slain the jabberwock
come to my arms my beamish boy
o frabjous day callooh callay
he chortled in his joy
twas brillig and the slithy toves
did gyre and gimble in the wabe
all mimsy were the borogoves
and the mome raths outgrabe
Using Regular Expressions¶
Find all words with a sequence of two or more identical letters e.g. “look”¶
import re
regex = re.compile(r'(\w*(\w)\2+\w*)', re.IGNORECASE | re.MULTILINE)
for match in regex.finditer(poem):
print(match.group(2), match.group(1))
b jabberwocky
l brillig
l all
b jabberwock
o took
e tree
o stood
f uffish
o stood
b jabberwock
f whiffling
o wood
b jabberwock
o callooh
l callay
l brillig
l all
Convert the identical sequences to uppercase¶
def f(match):
word, letter = match.groups()
return word.replace(letter, letter.upper())
print(regex.sub(f, poem))
jaBBerwocky
twas briLLig and the slithy toves
did gyre and gimble in the wabe
aLL mimsy were the borogoves
and the mome raths outgrabe
beware the jaBBerwock my son
the jaws that bite the claws that catch
beware the jubjub bird and shun
the frumious bandersnatch
he tOOk his vorpal sword in hand
long time the manxome foe he sought
so rested he by the tumtum trEE
and stOOd awhile in thought
and as in uFFish thought he stOOd
the jaBBerwock with eyes of flame
came whiFFling through the tulgey wOOd
and burbled as it came
one two one two and through and through
the vorpal blade went snickersnack
he left it dead and with its head
he went galumphing back
and hast thou slain the jaBBerwock
come to my arms my beamish boy
o frabjous day callOOh caLLay
he chortled in his joy
twas briLLig and the slithy toves
did gyre and gimble in the wabe
aLL mimsy were the borogoves
and the mome raths outgrabe
Natural language processing¶
If you intend to perform statistical analysis on natural language, you should probably use NLTK to pre-process the text instead of using string methods and regular expressions. For example, a simple challenge is to first parse the paragraph below into sentences, then parse each sentence into words.
Paragraph from random Pubmed abstract.
para = """When compared with the control group no significant associations were found for the NS-PEecl group after adjustment of confounding variables. For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity. Antiβ2GP1-domain I IgG were associated with aCL, antiβ2GP1 and aPS/PT IgG in the three groups. aPS/PT IgG were associated with aCL IgG, and aPS/PT IgM were associated with aCL and antiβ2GP1 IgM in the three groups CONCLUSION: S-PEecl is a distinct entity from NS-PEecl and is mainly associated with the presence of antiβ2GP1 IgG. Antiβ2GP1 domain I correlate with other aPL IgG tests, and aPS/PT may be promising in patients in which LA tests cannot be interpreted."""
Naive splitting of sentences as anything separated by ., ! or ?¶
sep = re.compile(r'[\?\!\.]')
ss = sep.split(para)
for i, s in enumerate(ss, 1):
print(i, ':', s, end='\n\n')
1 : When compared with the control group no significant associations were found for the NS-PEecl group after adjustment of confounding variables
2 : For the S-PEecl group, antiβ2GP1 IgG (OR 16
3 : 91, 95% CI 3
4 : 71-77
5 : 06) was associated, as well as age, obesity, smoking and multiparity
6 : Antiβ2GP1-domain I IgG were associated with aCL, antiβ2GP1 and aPS/PT IgG in the three groups
7 : aPS/PT IgG were associated with aCL IgG, and aPS/PT IgM were associated with aCL and antiβ2GP1 IgM in the three groups CONCLUSION: S-PEecl is a distinct entity from NS-PEecl and is mainly associated with the presence of antiβ2GP1 IgG
8 : Antiβ2GP1 domain I correlate with other aPL IgG tests, and aPS/PT may be promising in patients in which LA tests cannot be interpreted
9 :
Using NLTK¶
import nltk
ss_nltk = nltk.sent_tokenize(para)
for i, s in enumerate(ss_nltk, 1):
print(i, ':', s, end='\n\n')
1 : When compared with the control group no significant associations were found for the NS-PEecl group after adjustment of confounding variables.
2 : For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity.
3 : Antiβ2GP1-domain I IgG were associated with aCL, antiβ2GP1 and aPS/PT IgG in the three groups.
4 : aPS/PT IgG were associated with aCL IgG, and aPS/PT IgM were associated with aCL and antiβ2GP1 IgM in the three groups CONCLUSION: S-PEecl is a distinct entity from NS-PEecl and is mainly associated with the presence of antiβ2GP1 IgG.
5 : Antiβ2GP1 domain I correlate with other aPL IgG tests, and aPS/PT may be promising in patients in which LA tests cannot be interpreted.
Naive parsing of the second sentence into words¶
s = ss_nltk[1]
s
'For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity.'
# remove punctuation and split on whit space
table = dict.fromkeys(map(ord, string.punctuation))
s.translate(table).split()
['For',
'the',
'SPEecl',
'group',
'antiβ2GP1',
'IgG',
'OR',
'1691',
'95',
'CI',
'3717706',
'was',
'associated',
'as',
'well',
'as',
'age',
'obesity',
'smoking',
'and',
'multiparity']
Using NLTK¶
text = nltk.word_tokenize(s)
text
['For',
'the',
'S-PEecl',
'group',
',',
'antiβ2GP1',
'IgG',
'(',
'OR',
'16.91',
',',
'95',
'%',
'CI',
'3.71-77.06',
')',
'was',
'associated',
',',
'as',
'well',
'as',
'age',
',',
'obesity',
',',
'smoking',
'and',
'multiparity',
'.']
NLTK is a rich library for natural language processing¶
See http://www.nltk.org for details.
Tag tokens with part-of-speech labels¶
tagged_text = nltk.pos_tag(text)
tagged_text
[('For', 'IN'),
('the', 'DT'),
('S-PEecl', 'NNP'),
('group', 'NN'),
(',', ','),
('antiβ2GP1', 'RB'),
('IgG', 'NNP'),
('(', '('),
('OR', 'NNP'),
('16.91', 'CD'),
(',', ','),
('95', 'CD'),
('%', 'NN'),
('CI', 'NNP'),
('3.71-77.06', 'CD'),
(')', ')'),
('was', 'VBD'),
('associated', 'VBN'),
(',', ','),
('as', 'RB'),
('well', 'RB'),
('as', 'IN'),
('age', 'NN'),
(',', ','),
('obesity', 'NN'),
(',', ','),
('smoking', 'NN'),
('and', 'CC'),
('multiparity', 'NN'),
('.', '.')]
s
'For the S-PEecl group, antiβ2GP1 IgG (OR 16.91, 95% CI 3.71-77.06) was associated, as well as age, obesity, smoking and multiparity.'
A simplistic way to pick up nouns¶
[w for w, t in tagged_text if t.startswith('N')]
['S-PEecl',
'group',
'IgG',
'OR',
'%',
'CI',
'age',
'obesity',
'smoking',
'multiparity']
String formatting¶
Selection¶
import math
stuff = ('bun', 'shoe', ['bee', 'door'], 2, math.pi, 0.05)
'One: {}, Two {}'.format(*stuff)
'One: bun, Two shoe'
'One: {0}, Two {1}'.format(*stuff)
'One: bun, Two shoe'
'One: {1}, Two {1}'.format(*stuff)
'One: shoe, Two shoe'
'One: {0}, Two {2[1]}'.format(*stuff)
'One: bun, Two door'
Formatting¶
'One: {0:^10s}, Two {1:_>15s}'.format(*stuff)
'One: bun , Two ___________shoe'
'One: {3}, Two {4}'.format(*stuff)
'One: 2, Two 3.141592653589793'
'One: {3:+10d}, Two {4:.4f}'.format(*stuff)
'One: +2, Two 3.1416'
'One: {3:04d}, Two {4:.4g}'.format(*stuff)
'One: 0002, Two 3.142'
'One: {3:.4e}, Two {4:.4e}'.format(*stuff)
'One: 2.0000e+00, Two 3.1416e+00'
'One: {5:.2%}, Two {5:f}'.format(*stuff)
'One: 5.00%, Two 0.050000'
Old style formatting¶
'%s, %s, %a, %d, %.4f, %.2f' % stuff
"bun, shoe, ['bee', 'door'], 2, 3.1416, 0.05"
Formatting numpy arrays¶
import numpy as np
x = np.arange(1, 13).reshape(3,4)
x
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])
np.set_printoptions(formatter={'int': lambda x: '%8.2f' % x})
x
array([[ 1.00, 2.00, 3.00, 4.00],
[ 5.00, 6.00, 7.00, 8.00],
[ 9.00, 10.00, 11.00, 12.00]])
np.set_printoptions()
x
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])