UK National HPC Service

Computer Services for Academic Research Logo
Home | Helpdesk | Machine Status | Search | Apply
 

I/O and SAN performance issues

Introduction

In general, little is understood about the topic of I/O optimizationand it is often considered unimportant. However there are simple things that can be done to achieve better I/O performance at all levels whether it is copying large files, transferring data to our machines or running I/O intensive batch job. This page provides some useful pointers to increase the performance of I/O at all levels.

Optimizations

  • Moving and copying files - It is important to move and copy large numbers of files on the metadata server which is wren. The reason for this is that the overhead of moving a small file on another system is greater than on wren.
  • Avoid compiling codes on fermat and green - If you have a very large code the effect of compiling this code on green and newton will be noticeable and expensive.
  • Avoid using hold directly in a batch job - If the data is offline when it is requested it will take several minutes, in the best case, to retrieve it. Try to retrieve files interactively using wren and the dmget command or use cp -D to copy it to a more accessible location. This will save you significant time in your batch job.
  • Improving I/O performance within the code - There is often much that can be done here and if this is of serious concernthen you should contact CSAR who can provide more indepth help on your specific problems, but a few guiding principles are to avoid using ascii for large datasets as you pay a heavy penalty for the conversion into (or out of) ASCII format and will produce a far larger file, avoid doing small writes of a couple of elements at a time favouring the writing of large buffers as this leads to the ability to write this data directly to disk which can be achieved with FFIO in C codes and assign in Fortran codes (origin only).

Page maintained by This page last updated: Tuesday, 17-Aug-2004 15:32:56 BST