I/O and SAN performance issues
Introduction
In general, little is understood about the topic of
I/O optimizationand it is often considered unimportant. However there
are simple things that can be done to achieve better I/O performance
at all levels whether it is copying large files, transferring data to
our machines or running I/O intensive batch job. This page provides
some useful pointers to increase the performance of I/O at all levels.
Optimizations
- Moving and copying files -
It is important to move and copy large numbers of files on the metadata
server which is wren. The reason for this is that the overhead of moving
a small file on another system is greater than on wren.
- Avoid compiling codes on fermat and green -
If you have a very large code the effect of compiling this code on green
and newton will be noticeable and expensive.
- Avoid using hold directly in a batch job -
If the data is offline when it is requested it will take several minutes,
in the best case, to retrieve it. Try to retrieve files interactively
using wren and the dmget command or use cp -D to copy it to a more accessible
location. This will save you significant time in your batch job.
- Improving I/O performance within the code -
There is often much that can be done here and if this is of serious
concernthen you should contact CSAR who can provide more indepth help
on your specific problems, but a few guiding principles are to avoid
using ascii for large datasets as you pay a heavy penalty for the conversion
into (or out of) ASCII format and will produce a far larger file, avoid
doing small writes of a couple of elements at a time favouring the writing
of large buffers as this leads to the ability to write this data directly
to disk which can be achieved with FFIO in C codes and assign in Fortran
codes (origin only).
|
