Merging Concatenation and reshaping

1. Programming for Data
Analysis
Week 2
Dr. Ferdin Joe John Joseph
Faculty of Information Technology
Thai – Nichi Institute of Technology, Bangkok

2. Today’s lesson
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
2
• Merging
• Concatenating
• Reshaping
• Laboratory

3. Merging
• Used in pandas to combine data from two sources
• Sources can be from same format or different
• csv and csv, csv and json, json and xml and a concoction of all these
• Similar to numpy array manipulation but effective with pandas
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
3

4. Function Used
concat()
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
4

5. Syntax of concat()
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
5

6. Syntax of concat()
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
6
•objs : a sequence or mapping of Series or DataFrame objects. If a dict is passed, the sorted keys will be used as the keys argument,
unless it is passed, in which case the values will be selected (see below). Any None objects will be dropped silently unless they are all
None in which case a ValueError will be raised.
•axis : {0, 1, …}, default 0. The axis to concatenate along.

7. Syntax of concat()
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
7
•join : {‘inner’, ‘outer’}, default ‘outer’. How to handle indexes on other axis(es).
Outer for union and inner for intersection.
•ignore_index : boolean, default False. If True, do not use the index values on the
concatenation axis. The resulting axis will be labeled 0, …, n - 1. This is useful if you are
concatenating objects where the concatenation axis does not have meaningful indexing
information. Note the index values on the other axes are still respected in the join.

8. Syntax of concat()
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
8
•keys : sequence, default None. Construct hierarchical index using the passed
keys as the outermost level. If multiple levels passed, should contain tuples.
•levels : list of sequences, default None. Specific levels (unique values) to use
for constructing a MultiIndex. Otherwise they will be inferred from the keys.
•names : list, default None. Names for the levels in the resulting hierarchical index.

9. Syntax of concat()
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
9
•verify_integrity : boolean, default False. Check whether the new concatenated
axis contains duplicates. This can be very expensive relative to the
actual data concatenation.
•copy : boolean, default True. If False, do not copy data unnecessarily.

10. Example
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
10

11. Example
• Available Data frames: df1, df2 and df3
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
11

12. Creation of arrays
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
12

13. Creation of arrays
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
13

14. Creation of arrays
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
14

15. Creation of Arrays
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
15

16. Frames
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
16

17. Concatenation
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
17

18. Concatenation views
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
18

19. Setting other axes
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
19

20. Setting other axes
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
20

21. Setting other axes
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
21

22. Inner Join
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
22

23. Outer Join
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
23

24. Append()
• Alternative to concat()
• Combines two dataframes in first index only
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
24

25. Append
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
25

26. Sort
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
26

27. Append multiple dataframes
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
27

28. Varying dimension concatenation
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
28

29. Appending rows to a dataframe
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
29

30. How it works with csv, json and xml
• Convert these files to pandas dataframe object
• Play with concat or append
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
30

31. DSA 207 - Merging
• Create two arrays A1 and A2 and convert them into pandas data
frame. Merge the data frames and store in A2. Display A2 before and
after merging
• Merge the given csv files together using pandas and display the first
10 data and last 15 data.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
31