What are the most common performance problem causes?
From my personal experience (your mileage may vary):
* 50% poor disk layout + mgmt - some disks 90%+ busy while 50% not used at all - do you have a clear document that maps the files to actual disks?
* 10% poor setup of RDBMS tuning parameters relating to memory use
* 10% single threaded batch applications (and we have been using SMP for 9 years!!)
* 10% poorly written customer extensions to standard applications
* 5% system running with errors in the errpt log file (including CPU failures!!)
* 5% paging on large RAM (>2 GB) systems & vmtune not use to set min/maxperm
* 5% AIX problems already discovered and fixed but AIX was not up to date.
* 4% badly ported app = not compiled with optimization or on old AIX versions
* 1% genuine bugs in AIX or commands
and every single one of these was reported as a problem with the hardware!!!
Perfpmr - the performance guru's secret weapon
Perfpmr is the official AIX Support performance data gathering tool. If you report a performance problem then after the usual checks on your software levels, AIX Support will ask you to use this tool to gather loads of information that you must return, as soon as possible. It is actually a set of shell scripts that use standard tools to gather information and write report summaries. This means it is very useful for you too and why reinvent the wheel, when the IBM Austin experts are maintaining this excellent tool for you. Get the latest AIX version
It comes with an excellent README file with all the details on running it, but briefly, find at least 100 Mbytes of free disk space and as the root user untar the file and run the master perfpmr.sh shell script. This needs to run during the problem (of course) and I suggest for 5 minutes = 600 seconds. For example, ./perfpmr.sh 600 It actually takes longer than this as there are several sections and the final phase gathers lot of configuration details that takes a long time with machine with lots of disks. Once finished take a look at the summary (.sum) files as these are readable. You may be able to sort out the performance problem yourself from this data alone.
But perfpmr is more than this because:
* it was precisely documented your system configuration at the hardware and AIX levels - this saves you doing this.
* if you take regular perfpmr data of the system running happily on a normal workload day. Then when there is a problem - you or the AIX support team can take a look at the differences between the good and bad performance captures and this will make diagnosis ten times simpler.
* if you should take perfpmr data of the system running happily before and after any minor or major changes to the system. For example software upgrades, AIX upgrades, changes to disk subsystems and adding software. Again, you will be able to determine what changed - for good or bad - ten times faster with before and after data captures.
My favorite Desktop tools of the job
I have a Windows based Thinkpad and AIX servers and this is what I use every day:
1. Virtual Network Computing VNC from TightVNC - so I can have X Windows at zero cost and it stays running for tomorrow
2. WebSM Remote Client - so I can remotely manage my AIX machines and HMC's. You can download this from you HMC (if you have it setup right and allowed the protocol) from http://
5. Thrashing rising page outs, CPU wait and run queue.
6. Disk I/O bound when %iowait greater than 40% (iostat)
* Although in recent years this might not be true as CPUs are much faster and disks only a bit faster.
* This means many workloads are not disk bound and the CPUs deal with the data faster than the disks deliver it.
* So high I/O wait is perfectly normal on many systems.
Setting minfree and maxfree on an AIX 5.3 System
minfree = (maximum of either 960 or # of logical CPUs * 120) divided by # of memory pools
maxfree = minfree + (# of logical CPUs * maximum read ahead) divided by # of memory pools
Where,
1. of logical CPUs from bindprocessor -q (count number of available processors)
2. of memory pools from vmstat -v
(note: If the number is 0 use 1 as a default)
Maximum read ahead is the greater of maxpgahead or j2_maxPageReadAhead from ioo -a
Tuning minfree and maxfree
You need to increase the value of minfree when you see 'Free Frame Waits' increasing over time.
Use the 'vmstat -s' to display the currently value of 'Free Frame Waits'.
Remember, to calculate a new value for maxfree.
Large Disk Subsystem Setup
Setting up Disks? - Want some advice? Here are my (Nigel Griffiths) top tips:
1. More disks are goodness - increasingly hard to justify with bigger disks but spindles count for performance
2. Use ALL of the disks - ALL of the time. But make the computer do this i.e. not manually move data around.
3. Hose it all about (especially for systems with less than 8 disks = one RAID5 is best)
4. Using RAID5 - 7+1 parity for maximum disk use
5. Aim for 16 to 32 LUNs - make then big enough to reduce the number of LUNs - no one can manage a thousand LUN's
6. Use 4 paths = 64 to 128 vpaths - two paths also OK. Never use more than 4 paths.
7. All LUNs same size
8. Clear map of layout - you must known where LUNs are placed and overlap.
9. 4 to 16 filesystems (never just one filesystem) to avoid free space allocation bottlenecks.
10. With AIX 5.2 onwards, the 64 bit Kernel and JFS2 should be thought of as the default.
11. LVM striped at 64KB or 128KB stripe width.
12. Database should have 8KB or better yet 16KB block size minimum.
13. Mix data, index, logs up on all the disks - to avoid hot spots (don't have disks for specific data types)
14. Direct attach, if you are the only ESS/FAStT user
15. Sequential I/O - big block = max throughput above 64KB blocks, also 256KB or 1MB works well (4KB can kill your throughput)
16. Random I/O - many files, equally spread across filesystems and hence disks
17. Two Fibre Channel adapters per TB,
18. Expect 70% adapter bandwidth max (2 Gbit FC = max 200MB/s and 140MB/s real)
For ESS/FAST/DS in particular (these numbers are 2 years old and probably need updating):
1. 6 to 8 Fibre Channel adapters per ESS
2. Disk Size: 36 GB=OK, 72 GB OK for Random, 146 GB only for archive
3. Use ESS bid against EMC and HDS
4. Typically 30+ hosts
5. Disk qdepth ESSF20=60 and ESS800=90 FAST900=1024/disks/hosts FAST500+600= 212/hosts/disks
6. Fibre Channel memory DMA=1MB, CMD's 248
Thursday, July 24, 2008
Performance Monitoring Tips and Techniques - AIX
Kenapa Gaji Saya Tidak Pernah Cukup
Gaji. Cuma 4 huruf memang. Tapi kenyataannya, yang namanya gaji selalu saja ditunggu-tunggu oleh mereka yang bekerja sebagai karyawan. Dimanapun itu.
Betul. Jika Anda - atau suami Anda - bekerja sebagai karyawan di sebuah perusahaan, seringkali Anda selalu menunggu agar tanggal gajian itu datang - kalau bisa - sepuluh hari lebih cepat. Benar kan?
Menariknya, berapa pun gaji yang terima, seringkali kita merasa tidak pernah cukup. Besar sih besar. Itu sebabnya, kadang kita merasa kesal, kenapa sudah capek-cepek kerja sebulan, tetapi gaji tak pernah cukup.
Bapak ibu, tenang. Kalau Anda bingung karena merasa gaji Anda tidak pernah cukup, maka kali ini saya akan memberitahu 5 hal kenapa gaji Anda dirasa tidak pernah cukup. Mau tahu apa saja?
1. Biaya Hidup Tinggi
Salah satu penyebabnya mungkin karena biaya hidup di kota tempat Anda tinggal memang tinggi, dibandingkan kota-kota lain. Sekali lagi, ini cuma kemungkinan saja lho.
2. Gaji Anda Terlalu Kecil
Penyebab kedua adalah gaji yang kecil. Mungkin biaya hidup di kota Anda tak terlalu tinggi, tapi berhubung gaji Anda kecil, jelas Anda merasa gaji tak cukup bukan? Memang ada yang berpendapat, berapa pun gaji Anda, asal bisa dikelola dengan baik, pasti cukup. Betul. Saya setuju, karena saya sendiri juga bilang begitu. Tapi, kalau Anda merasa telah mengelolanya dengan baik, tapi kok gaji Anda tetap saja dirasa enggak pernah cukup selama berbulan-bulan, bahkan bertahun-tahun, maka bisa jadi gaji Anda memang terlalu kecil.
3. Kenaikan Gaji tak Sebanding dengan Kenaikan Biaya Hidup
Penyebab ketiga karena kenaikan gaji Anda setiap tahun tidak seimbang dengan kenaikan Biaya Hidup. Misalnya, gaji di keluarga Anda Rp 1 juta sebulan. Biaya hidup Anda sekeluarha juga Rp 1 juta sebulan. Sementara tahun depan harga-harga naik, sehingga biaya hidup misalnya Rp 1,3 juta sebulan. Semetara gaji Anda "hanya" naik 10 persen, menjadi Rp 1,1 juta. Jelas, kan, gaji Anda tidak cukup.
3. Uang Lebih Enak Dihabiskan daripada Ditabung
Jujur saja, dimana-mana, yang namanya uang lebih enak untuk dihabiskan daripada ditabung. Benar kan? Contohnya, kalau Anda dapat uang Rp 500 rb sekarang, pasti akan lebih enak kalau Anda membelanjakannya daripada disimpan. Bawa saja uang itu ke toko baju atau toko sepatu, pilih-pilih, beli deh.
Tapi, kalau ditabung, enggak enak rasanya. Kenapa? Karena Anda jadi enggak bisa belanja. Itulah kenapa yang namanya uang lebih enak dibelanjakan daripada ditabung. Nah, karena keenakan, membuat orang selalu menghabiskan uang di dompetnya.
Setiap kali Anda pegang uang, selalu habis. Akhirnya, ketika ada kebutuhan, Anda lantas berpikir: "Kok nggak cukup ya gaji saya?" Padahal, itu karena Anda sudah terlanjur keenakan menghabiskan uang, sehingga rasanya gaji Anda seperti tidak cukup. Padahal, mungkin saja gaji Anda sebetulnya cukup, tapi Anda terlalu menghambur-hamburkannya.
4. Karena Keinginan Tidak Ada Batasnya
Anda tahu tidak bedanya kebutuhan dan keinginan? Kebutuhan adalah sesuatu yang Anda beli karena memang Anda butuhkan, entah Anda ingin atau tidak. Sementara keinginan, adalah sesuatu yang Anda beli, walaupun kadang Anda tidak selalu membutuhkannya. Menariknya, kebutuhan seringkali memiliki batas. Sementara keinginan biasanya tidak memiliki batas.
Misalnya, setiap bulan, penghasilan keluarga Anda katakan saja Rp 1 juta. Biaya hidup yang betul-betul Anda butuhkan setiap bulan rata-rata adalah Rp 700 ribu (ada batasnya, ingat?). Sementara, barang-barang yang Anda beli karena Anda memang menginginkannya, rata-rata adalah Rp 300 ribu per bulan. Total Rp 1 juta. Sesuai dengan gaji Anda. Tapi, suatu kali, bisa saja keinginan Anda meningkat, dari yang biasanya Rp 300 ribu per bulan, menjadi Rp 500 ribu per bulan. Otomatis, penghasilan Anda jadi nggak cukup dong.
Menariknya, seringkali justru keinginan inilah yang membuat gaji kita terasa tidak cukup, karena yang namanya keinginan seringkali tidak ada batasnya, sementara kebutuhan, umumnya ada batasnya (dalam contoh diatas tadi, yaitu Rp 700 ribu per bulan). Nah, kalau Anda ingin gaji Anda cukup, sering-seringlah mengendalikan keinginan Anda yang biasanya tidak terbatas itu.
Bagaimana bapak ibu. Sudah jelas kan kenapa Anda selalu merasa gaji Anda tidak pernah cukup? Salam.
