dNFS for DBA's

© 2017 Delphix. All Rights Reserved. Private and Conﬁdential.
Marcin Przepiorowski | Senior Technical Principal| November 2017
dNFS for DBA’s

Oracle consultant/DBA since 2000

co-developer of OraSASH – free ASH/AWR like
repository

Blogger ???

About me

Delphix Dynamic Data Platform
Personal ﬂexible
data pods for
everyone

4
Network
Conﬁguration
Examples

Michael Coles

https://pixabay.com/en/water-hose-garden-wet-gardening-815475/

8 Gb Fiber Channel = 10 Gb Ethernet NFS
Network

Throughput is one dimension to measure.

Latency is even more important.

Latency has impact on:
–  single block reads
–  real throughput

Network

100 m
Dog can run 70 km/h – 100 m in 5 sec
Dog can carry a 2 TB SSD drive

Throughput = 2 * 1024 / 5 sec = 409 GB/s
Latency – 5 sec
RTT – 10 sec
Other – RFC 1149 IP over Avian Carriers

Network
Recommended latency for (d)NFS < 1 ms
Jumbo Frames enabled

1 TCP stream vs multiple TCP streams
Network

http://www.ishn.com/ext/resources/todaysnews3/traﬃc-422.jpg
http://www.livemint.com/rf/Image-621x414/LiveMint/Period1/2015/08/11/Photos/
traﬃc-kjXE--621x414@LiveMint.jpg
1 TCP stream

vs multiple TCP streams

Download Managers
Network

NFS opens one TCP stream
dNFS opens multiple TCP streams
Network

14
Conﬁguration
Network
Examples

•  dNFS support >=11g
•  Unix/Linux, Windows
•  Using ODM for file system calls
•  Talking directly to filer
Direct NFS
File Type Supported
Control file YES
Data file YES
Redo log file YES
Archive/Flashback log file YES
Backup files YES
Temp file YES
Datapump dump file YES
OCR files NO
spfile YES
passwd file YES
ASM files YES
Voting files NO
Audit files NO
Database trace files NO
External tables YES (12c)

Enable

$ cd $ORACLE_HOME/rdbms/lib/
$ make -f ins_rdbms.mk dnfs_on
rm -f /u01/app/oracle/11.2.0.4/db1/lib/libodm11.so; cp /u01/app/oracle/
11.2.0.4/db1/lib/libnfsodm11.so /u01/app/oracle/11.2.0.4/db1/lib/libodm11.so

Disable

$ make -f ins_rdbms.mk dnfs_off
rm -f /u01/app/oracle/11.2.0.4/db1/lib/libodm11.so; cp /u01/app/oracle/
11.2.0.4/db1/rdbms/lib/libodm11.so.dummy /u01/app/oracle/11.2.0.4/db1/lib/
libodm11.s
dNFS conﬁguration – 11g

Enable

$ make -f ins_rdbms.mk dnfs_on
rm -f /u01/app/oracle/12.1.0.2/rachome1/rdbms/lib/odm/libnfsodm12.so;
cp /u01/app/oracle/12.1.0.2/rachome1/lib/libnfsodm12.so /u01/app/oracle/
12.1.0.2/rachome1/rdbms/lib/odm/libnfsodm12.so

Disable

$ make -f ins_rdbms.mk dnfs_off
rm -f /u01/app/oracle/12.1.0.2/rachome1/rdbms/lib/odm/libnfsodm12.so
dNFS conﬁguration – 12g

•  There is no configuration required for basic use case.
•  Configuration file oranfstab is optional
•  But sometimes ALTER DATABASE MOUNT ends up with
ORA-00600: internal error code
UEK Kernel bug ID 1460787.1
dNFS configuration

$ORACLE_HOME/dbs/oranfstab
/etc/oranfstab
/etc/mtab
 
server: Delphix
# This is only name 
path: 192.168.166.141
# IP of NFS server 
local: 192.168.166.142
# IP of interface on DB server 
export: /oraclenfs mount: /oradata1
# mount points

dNFS conﬁguration

•  dNFS is looking for each data file path in oranfstab
•  If not found, information from /etc/mtab is used
•  Multipath configuration requires an entry about all file systems in oranfstab
•  File systems without matching entry won’t use multipath

dNFS configuration

Multipath
server:DE
local:172.16.169.142 path:172.16.169.141
local:192.168.166.24 path:192.168.166.23
export:/orc_timeflow-79/datafile mount:/mnt/provision/SLOB/datafile
export:/orc_timeflow-79/temp mount:/mnt/provision/SLOB/temp
export:/orc_timeflow-79/archive mount:/mnt/provision/SLOB/archive
dNFS conﬁguration

oradism

RAC (LMS scheduling),
Solaris “Dynamic Intimate Shared Memory”

$ORACLE_HOME/bin/oradism
ls -l $ORACLE_HOME/bin/oradism
-rwsr-x--- 1 root oinstall ……

dNFS conﬁguration – 11.2.0.4
-rwsr-x

-rwxr-x--- 1 oracle oinstall ……
Wed Mar 22 16:49:15 2017
ALTER DATABASE MOUNT
Direct NFS: please check that oradism is setuid

Wed Aug 30 21:20:03 2017
Direct NFS: please check that oradism is setuid
…
Completed: ALTER DATABASE MOUNT
…
Direct NFS: channel id [0] path [172.16.180.135] to filer
[172.16.180.135] via local [] is UP
Wed Aug 30 21:20:07 2017
Direct NFS: channel config is:
Wed Aug 30 21:20:07 2017
channel id [0] local [] path [172.16.180.135]


21:27:26 SYS@siclone:>select * from v$dnfs_servers;
no rows selected


open("/mnt/xxx/RACDBA/DATAFILE/system.278", O_RDWR|O_DSYNC|O_DIRECT) = 8
fcntl(8, F_DUPFD, 256)= 256
io_submit(140452885917696, 1, {{data:0x7ffc54a405a8, preadv) = 1
pread(256, "202420"..., 8192, 24641536) = 8192


-rwsr-x--- 1 oracle oinstall ……

21:43:37 SYS@siclone:>select id, svrname, dirname from v$dnfs_servers;
ID SVRNAME DIRNAME
-- -------------- --------------------------------------------------------
1 172.16.180.135 /oracle_db_container-1506/oracle_timeflow-1716/datafile
1 172.16.180.135 /oracle_db_container-1506/oracle_timeflow-1716/temp


connect(32, {…,sin_port=htons(2049),sin_addr=inet_addr("172.16.180.135")},
sendmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{“1"..., 152}],
poll([{fd=32, events=POLLIN}], 1, 500) = 1 ([{fd=32, revents=POLLIN}])
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{

•  MOS
–  Recommended Patches for Direct NFS Client (Doc ID 1495104.1)
•  11.2.0.4 and 12.1.0.2 looks OK
–  How To Setup DNFS (Direct NFS) On Oracle Release 11.2 (Doc ID 1452614.1)
–  Step by Step - Configure Direct NFS Client (DNFS) on Linux (11g) (Doc ID 762374.1)
–  Step by Step - Configure Direct NFS Client (DNFS) on Windows (Doc ID 1468114.1)
•  Blogs
–  http://blog.oracle48.nl/wordpress/direct-nfs-configuring-and-network-considerations-in-practise/
–  http://www.slideshare.net/yvelikanov/sharing-experience-implementing-direct-nfs#

dNFS configuration

•  TCP stack parameters
–  Oracle recommend increase buffers to 4 MB
–  This is not enough for fast LAN’s (10 Gb)
–  Other settings are recommended as well

•  For fast LAN network 16 MB buffers can be fully utilized
•  Some systems require a change of NFS block size

dNFS configuration
https://docs.oracle.com/database/122/LADBI/checking-tcp-network-protocol-buffer-for-direct-nfs-client.htm

Linux ( Red Hat >= 6.3 )

net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 16777216 16777216
net.ipv4.tcp_wmem = 4096 4194304 16777216
dNFS configuration
https://docs.delphix.com/docs/system-administration/performance-tuning-configuration-and-analytics/target-host-os-and-database-configuration-options

Solaris 11

ipadm set-prop -p max_buf=16777216 tcp
ipadm set-prop -p _cwnd_max=4194304 tcp
ipadm set-prop -p send_buf=4194304 tcp
ipadm set-prop -p recv_buf=16777216 tcp

NFS block size – change from 32 kB to 1MB
/etc/system
set nfs:nfs3_bsize=0x100000

dNFS conﬁguration

AIX
Disable delayed ACK
tcp_nodelayack=1

NFS
nfs_max_read_size=524288
nfs_max_write_size=524288

Fix for 64k NFS block size
6.1
IV24594
7.1
IV24688

dNFS conﬁguration

NFS

nfsiostat, netstat, ss, wireshark

dNFS

v$dnfs_stats, v$dnfs_channels

netstat, ss, wireshark
Monitoring

36
Examples
Network
Conﬁguration

0.00
10,000.00
20,000.00
30,000.00
40,000.00
50,000.00
60,000.00
70,000.00
80,000.00
1
2
4
8
16
24
32
40
48
IOPS
Number of SLOB processes
IOPS
dNFS
dNFS
NFS
NFS

IOPS – response time – 40 processes
1.0 ms
0.61 ms

40 k IOPS
IOPS CPU utilization
dNFS
NFS

FILESYSTEMIO_OPTIONS

- SETALL

- NONE / ASYNCH

NFS can use OS cache depend on value of parameter
dNFS is not using OS ﬁle system and OS cache
Impact of Oracle parameters

0.00
100,000.00
200,000.00
300,000.00
400,000.00
500,000.00
600,000.00
700,000.00
1
2
4
8
16
24
32
40
48
IOPS
Number of SLOB processes
IOPS
NFS big dataset
NFS small dataset
dNFS big dataset
dNFS small dataset

INVESTIGATION

Real life example
Block #2: noparallel FULL table scan
NFS
dNFS
Stat: query - elapsed (s) :
57.43
146.04
Stat: query - row count :
648,300
648,300
Stat: query - physical reads (MB/s) :
176.38
69.362
Block #3: parallel FULL table scans…
Stat: parallel4 - MB/s :
179.43
260.75
177.24
450.42
173.45
459.39
171.98
453.24

Examples - NFS - RedHat 6.8 UEK 3.8.13
Block #2: noparallel FULL table scan
39.49
5,067,786
1,002.59
Block #3: parallel FULL table scans...
1,174.03
1,177.87
1,173.68
1,176.12

Examples - dNFS - RedHat 6.8 UEK 3.8.13
Block #2: noparallel FULL table scan...
74.71
5,067,786
529.943
Block #3: parallel FULL table scans...
1,013.95
1,152.86
1,169.18
1,174.38

NFS
Tota Wait % DB
Event Waits Time Avg(ms) time Wait Class
------------------------------ ------------ ---- ------- ------ ----------
direct path read 384,380 1341 3 97.4 User I/O
DB CPU 35.1 2.5
dNFS
Tota Wait % DB
Event Waits Time Avg(ms) time Wait Class
------------------------------ ------------ ---- ------- ------ ----------
direct path read 164,147 1402 9 98.8 User I/O
DB CPU 119. 8.4
Example – table full scan

•  Top function : copy_user_generic_unrolled
•  It used when there is no optimization on CPU level
•  Fast string operations ( Enhanced REP MOVSB/SROSB) are unsupported.
Linux version 3.8.13-118.14.1.el6uek.x86_64 (mockbuild@x86-ol6-builder-04)
(gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #2 SMP Mon Oct 31
17:33:13 PDT 2016
Command line: ro root=/dev/mapper/vg_dnfstargetdb1-lv_root rd_NO_LUKS
Disabled fast string operations
Investigation

•  perf record -g -F9999 -p 6101
•  perf script | gzip > dnfs2.txt.gz
•  stackcollapse-perf.pl dnfs2.txt > dnfs2_1.folded
•  ﬂamegraph.pl dnfs2_1.folded > dnfs2_1.svg
Investigation

OS call
OS call
Top function

http://www.newsweek.pl/biznes/animacja-reklamy,artykuly,41747,1,1,1.html

NFS
LADDR LPORT RADDR RPORT SWND CWND RWND
192.168.166.23 2049 192.168.166.24 971 8379904 1744860 4196612
dNFS
192.168.166.23 2049 192.168.166.24 36189 71608 44740 4196612
192.168.166.23 2049 192.168.166.24 48924 247052 16777216 4196612
192.168.166.23 2049 192.168.166.24 50202 247052 16777216 4196612
192.168.166.23 2049 192.168.166.24 38596 247052 16777216 4196612
Investigation - Network


“bandwidth-delay product refers to the product of a data link's capacity "
(in bits per second) and its round-trip delay time (in seconds)”
https://en.wikipedia.org/wiki/Bandwidth-delay_product

If

BDP (in bytes) > TCP window

then the TCP session will not be able to use all of the available
bandwidth

Bandwidth = (window size *8)/RTT

0
100000
200000
300000
400000
500000
600000
1
6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
101
106
111
116
121
126
131
136
Bytes
Bytes in ﬂight
NFS
dNFS

TCP settings on OS level are some for both runs
and TCP windows can be as big as 16M

net.ipv4.tcp_rmem = 4096 16777216 16777216
net.ipv4.tcp_wmem = 4096 4194304 16777216

strace output of Oracle process

setsockopt(32, SOL_SOCKET, SO_SNDBUF, [262144], 4) = 0
setsockopt(32, SOL_SOCKET, SO_RCVBUF, [262144], 4) = 0
bind(32, {sa_family=AF_INET, sin_port=htons(0),
sin_addr=inet_addr("192.168.166.24")}, 16) = 0
connect(32, {sa_family=AF_INET, sin_port=htons(2049),
sin_addr=inet_addr("192.168.166.23")}, 16) = 0
setsockopt(32, SOL_SOCKET, SO_SNDBUF, [1056768], 4) = 0
setsockopt(32, SOL_SOCKET, SO_RCVBUF, [1056768], 4) = 0

•  Kernel allocate 2 x SO_RCVBUF for overhead
•  Parameter tcp_adv_win_scale is controlling an overhead buffer 
Default value is 1

TCP Window = buffer - buffer/2^tcp_adv_win_scale

TCP Window = 512k – 256/2^1 = 256k

https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt, http://man7.org/linux/man-pages/man7/socket.7.html

tcp_adv_win_scale = 2
Stat: query - physical reads (MB/s) : 721.704
192.168.166.23 2049 192.168.166.24 34770 375056 16777216 4196612
192.168.166.23 2049 192.168.166.24 46897 375056 16777216 4196612
tcp_adv_win_scale = 4
Stat: query - physical reads (MB/s) : 712.985
192.168.166.23 2049 192.168.166.24 44832 471056 6970492 4196612
192.168.166.23 2049 192.168.166.24 39377 471056 2925996 4196612

•  dNFS is opening a connection(s) per process
•  Socket buﬀers are allocated by process inside kernel space
•  Total memory is 2 x buﬀer size ( one part is for administration and kernel internal structures)
dNFS memory
60

Investigation - Latency
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
0
0.1
0.2
0.4
0.5
1
2
%
Additional latency
% of max throughput
NFS
dNFS
tc qdisc add dev eth1 root netem delay Xms


It depends

Run a test with your workload
and network

Who is a winner ?

Marcin Przepiorowski
@pioro
marcin@delphix.com
Thank you for attending my session
Q&A

dNFS for DBA's

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to dNFS for DBA's

Similar to dNFS for DBA's (20)

Recently uploaded

Recently uploaded (20)

dNFS for DBA's