Multipath I/O and Network bonding

Event though Multipath I/O and NIC bonding is more related to sysadmin and storage admin tasks, it is good idea to grab some knowledge about those keywords as they play important role in achieving good performance.

In the series of articles about disk speed I explained why it is good to know what values you can expect for disk speed, there is one thing I didn’t mention which is critical when you connect your database server to your enterprise SAN storage and that is Multipath I/O.

Multipath I/O has the following two objectives:

• redundancy (as more than one path between server and SAN storage exist)
• performance enhancement

Essentially only one simple command is enough to check if Multipath I/O has been configured or not:

multipath -ll

As Oracle DBA/Developer it is enough to know two essential information:

• to confirm that multipath is configured and enabled
• what are your Enterprise storage speed limitations

Point is: if multipath is not configured and started (which can be seen either by executing command from above or indirectly from some of the disk monitoring utilities or Oracle AWR reports), all you need to do is to check with sysadmins to configure it properly and your job is over.

Second part of this article is related to network card(s) configuration and bonding.

Network speed is one of the critical components of your system for several reasons:

• Your database servers need fast communication with enterprise storage.
• In case of RAC, fast communication among RAC Node is essential for cluster stability and RAC cache Fusion (case when user connected through first node queries blocks that are in buffer cache on some other nodes, where other RAC instance will deliver cached blocks through the cluster interconnect).
• Your integration servers need fast connection (for example Oracle Service Bus) as such servers are connected with many end points.
• Application servers need to have good link with Database servers and your client machine, as end user always expect to get results very fast.
• If you are running part of your infrastructure in Cloud, it’s important to measure network performance between your data center (on premise part of infrastructure) and part that is in the Cloud.

To measure network speed, there are several utilities you can use. Here I’ll use iperf.

On one side (server side) you need to open port 5001 and to run:

iperf -s

On the other side (client side) you can execute something like this:

iperf -c 192.168.8.110
------------------------------------------------------------
Client connecting to 192.168.8.110, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.8.101 port 56726 connected with 192.168.8.210 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   558 MBytes   468 Mbits/sec

More realistic way is to execute the following command on the server side:

iperf -l 1M -w 4M -f M -s

and run the following command on the other side:

iperf -l 1M -w 4M -f M -t 60 -c 192.168.8.110
------------------------------------------------------------
Client connecting to 192.168.8.110, TCP port 5001
TCP window size: 0.41 MByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[  3] local 192.168.8.101 port 56758 connected with 192.168.8.210 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  3344 MBytes  55.7 MBytes/sec

Short description of various switches:

• l —> read/write buffer (here we have 1MB buffer)
• w —> TCP window size (here it’s 4MB)
• f —> display results in MB/sec, not Mbits/sec
• t —> duration of testing network speed (60 sec in this case)

As we can observe, max. network throughput is 55.7 MB/sec on my home network, where two servers are connected through Ethernet (and home router between those server).

If I repeat the same test over wi-fi on both sides, I’ve got the following results:

Server side: 
[server]# iperf -l 1M -w 4M -f M -s

Client side: 
iperf -l 1M -w 4M -f M -t 60 -c 192.168.8.211
------------------------------------------------------------
Client connecting to 192.168.8.211, TCP port 5001
TCP window size: 0.41 MByte (WARNING: requested 4.00 MByte)
------------------------------------------------------------
[  3] local 192.168.8.200 port 40354 connected with 192.168.8.211 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec   248 MBytes  4.13 MBytes/sec

Network speed in the later (wifi) case has dropped from 55.7 MB/sec to 4.13 MB/sec which is more than 13 times.

If you find your server network capacity is not enough, you can add more network cards in your server or you can redistribute existing network cards and bond them together to solve your issue.

Summary:

There are many moving parts in today’s architecture that need some care and knowledge to configure it properly.

If you are not taking care about all components, in high demand applications (like Oracle Retail) you can expect to have end user disappointment and many incidents.

Even if you are using Cloud infrastructure or engineered systems (like Oracle’s Exadata), you are not safe, as such systems might also be inappropriate for your specific case until you perform some tests and reconfiguration or add some resources.

---