Vectorized String Operations

1. INTRODUCING PANDAS STRING OPERATIONS

import numpy as np
import pandas as pd 

Vectorization is process of doing an operation on multiple items (in an array, for example) in one go.

x  = np.array([1,2,3,4,5])

# performing vectorization of operations
x * 10

array([10, 20, 30, 40, 50])

However, it is not straightforward to perform vectorization on “array of strings” and Pandas addresses this need of performing vectorized string operations using various str methods

names_series  = pd.Series(['tom','JOhn','MARIA'])
names_series

0      tom
1     JOhn
2    MARIA
dtype: object

names_series.str.capitalize()

0      Tom
1     John
2    Maria
dtype: object

2. STRING OPERATIONS

Let’s first define a Pandas Series to work with:

# Panda series use in this section
names = pd.Series(['Walter White', 'Jesse Pinkman', 'Skyler White', 'Hank Shrader', 'Mike Ehrmantraut', 'Gus Fring']) 
names

0        Walter White
1       Jesse Pinkman
2        Skyler White
3        Hank Shrader
4    Mike Ehrmantraut
5           Gus Fring
dtype: objec

2.1 Methods Similar to Python String Methods

Nearly all Python’s built-in string methods are mirrored by a Pandas vectorized string method. Visit this link to get the complete list.

# lets apply some of these string methods to panda series
# to upper case
names.str.upper()

0        WALTER WHITE
1       JESSE PINKMAN
2        SKYLER WHITE
3        HANK SHRADER
4    MIKE EHRMANTRAUT
5           GUS FRING
dtype: objectt

# to check if it is digit
names.str.isdigit()

0    False
1    False
2    False
3    False
4    False
5    False
dtype: bool

# to get length of each item in the array
names.str.len()

0    12
1    13
2    12
3    12
4    16
5     9
dtype: int64

# to get boolean array, one that passes the condition
names.str.startswith('W')

0     True
1    False
2    False
3    False
4    False
5    False
dtype: bool

2.2. String Methods using Regular Expressions

Regular expression is a special syntax to find string or set of strings. This topic is very broad and can be very dry. However, we are going to taste plain-vanilla flavor of them here.

The following methods accept regular expressions to examine the content of each string element, and follow some of the API conventions of Python’s built-in re module

# let apply str.extract() method with regular expression to extract the first names
names.str.extract('([A-Za-z]+)')

        0
0  Walter
1   Jesse
2  Skyler
3    Hank
4    Mike
5     Gus

There are some good introductory examples on regular expressions usage in Python here

2.3. Vectorized indexing and slicing

# getting first letter of each element in the array
# using standard indexing method
names.str[0]

0    W
1    J
2    S
3    H
4    M
5    G
dtype: object

# getting first letter of each element in the array
# using  str.get() method
names.str.get(0)

0    W
1    J
2    S
3    H
4    M
5    G
dtype: object

# str.slice()
names.str.slice(0,2)

0    Wa
1    Je
2    Sk
3    Ha
4    Mi
5    Gu
dtype: object

# str.split()
names.str.split()

0        [Walter, White]
1       [Jesse, Pinkman]
2        [Skyler, White]
3        [Hank, Shrader]
4    [Mike, Ehrmantraut]
5           [Gus, Fring]
dtype: object

# str.split() with str.get(0) to get first name
names.str.split().str.get(0)

0    Walter
1     Jesse
2    Skyler
3      Hank
4      Mike
5       Gus
dtype: object

2.4. get_dummies

The get_dummies() lets you quickly split out indicator variables into a DataFrame

dummy = pd.DataFrame({'info': ['A|B|C','A','A|C'],
                     'name': ['tom','dick','harry']})
print(dummy)

    info   name
0  A|B|C    tom
1      A   dick
2    A|C  harry

# using get_dummies
print(dummy['info'].str.get_dummies('|'))

   A  B  C
0  1  1  1
1  1  0  0
2  1  0  1