The Parra Cluster at Univ. de Santiago de Chile

During a ten day visit at Santiago, in the context of an international project supported by Fundación Andes (Chile), we built a four node cluster. In the installation participated Marcela Cruchaga and Diego Celentano from USACH and Norberto Nigro and Mario Storti from CIMEC (INTEC-CONICET-UNL).

The cluster nodes have hard-disks and some notes of the installation follow (mostly in spanish):

• 4 Procesadores Pentium IV 128Mb Rambus RAM 600 MHz Samsung. See ProcessorPerformance for some details about the Pentium 4/Rambus memory combination
• Placa madre Intel 850MV
• Se instalo Red Hat 7.1 con configuracion Personalizada
• Instalar paquetes blas y lapack
• Desactivar el modo grafico en /etc/inittab poniendo un 3 en el runlevel por default es decir `id:3:initdefault:'
• El directorio /home es montado via NFS desde los nodos al server, de manera que la cuenta de los usuarios es la misma en todos los nodos y server. Para eso un dado usuario tiene que tener el mismo `uid' y `gid' en todas las maquinas.

Network configuration

• In the server

• • /etc/sysconfig/network-scripts/ifcfg-lo:

DEVICE=lo
IPADDR=127.0.0.
NETMASK=255.0.0.0
NETWORK=127.0.0.0
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
BOOTPROTO=none

• • /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE="eth0"
BOOTPROTO="none"
BROADCAST=10.0.0.255
IPADDR="10.0.0.1"
NETMASK="255.255.255.0"
NETWORK=10.0.0.0
ONBOOT="yes"

• • /etc/sysconfig/network:

NETWORKING=yes
HOSTNAME="parra1"
DOMAINNAME=cluster
FORWARD_IPV4="no"

• • /etc/hosts:

127.0.0.1	localhost.localdomain	localhost
10.0.0.1	parra1.cluster	parra1
10.0.0.2	parra2.cluster	parra2
10.0.0.3		  parra3.cluster  parra3
10.0.0.4		  parra4.cluster  parra4

• • /etc/exports:

/home  parra2.cluster (rw,no_all_squash,no_root_squash)
/usr/local parra2.cluster (rw,no_all_squash,no_root_squash)

/home  parra3.cluster (rw,no_all_squash,no_root_squash)
/usr/local parra3.cluster (rw,no_all_squash,no_root_squash)

/home  parra4.cluster (rw,no_all_squash,no_root_squash)
/usr/local parra4.cluster (rw,no_all_squash,no_root_squash)

• • /etc/hosts.equiv:

parra1.cluster
parra1  # puede o no puede estar (?????)
10.0.0.1

parra2.cluster
parra2
10.0.0.2

parra3.cluster
parra3
10.0.0.3

parra4.cluster
parra4
10.0.0.4

• • /etc/hosts.allow:

#
# hosts.allow	This file describes the names of the hosts which are
#		allowed to use the local INET services, as decided
#		by the '/usr/sbin/tcpd' server.
#

ALL: localhost
ALL: 127.0.0.1

ALL: 10.0.0.1
ALL: parra1.cluster

ALL: 10.0.0.2
ALL: parra2.cluster

ALL: 10.0.0.3
ALL: parra3.cluster

ALL: 10.0.0.4
ALL: parra4.cluster

• • /etc/hosts.deny: existente pero vacio

• • /etc/networks:

cluster 10.0.0.0

• • /etc/resol.conf: existente / vacio (muy importante!!!)

• • /etc/host.conf:

order hosts,bind

• En los nodos (por ejemplo el nodo 2)

• • /etc/sysconfig/network-scripts/ifcfg-lo

DEVICE=lo
IPADDR=127.0.0.2
NETMASK=255.0.0.0
NETWORK=127.0.0.0
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
BOOTPROTO=none

• • /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=none
BROADCAST=10.0.0.255
IPADDR=10.0.0.2
NETMASK=255.255.255.0
NETWORK=10.0.0.0
ONBOOT=yes

• • /etc/sysconfig/network:

NETWORKING=yes
HOSTNAME=parra2
DOMAINNAME=cluster
FORWARD_IPV4=yes

• • /etc/hosts:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1		localhost.localdomain localhost
10.0.0.1	  parra1.cluster  parra1
10.0.0.2	  parra2.cluster  parra2
10.0.0.3	  parra3.cluster  parra3
10.0.0.4	  parra4.cluster  parra4

• • /etc/exports:

• • /etc/hosts.equiv: igual al del server

• • /etc/hosts.deny: existente pero vacio

• • /etc/networks:

cluster 10.0.0.0

• • /etc/fstab: agregar las lineas

parra1.cluster:/home			 /home				  nfs	  hard,intr,rw
parra1.cluster:/usr/local	  /usr/local			nfs	  hard,intr,r

• • /etc/resolv.conf: debe estar vacio (muy importante!!!)

• Resultados de correr el programa `nettest'

/usr/local/mpi/bin/mpirun  -np 4 -machinefile machi.dat nettest.bin
Results for chunk_size = 100000 double, (800000 bytes, 6400000 bits) ----------
proc 0 <-> proc 1 connection bandwidth: 91.7584 Mbit/sec
proc 0 <-> proc 2 connection bandwidth: 89.5303 Mbit/sec
proc 0 <-> proc 3 connection bandwidth: 91.8042 Mbit/sec
proc 1 <-> proc 2 connection bandwidth: 89.2515 Mbit/sec
proc 1 <-> proc 3 connection bandwidth: 89.1865 Mbit/sec
Results for chunk_size = 10000 double, (80000 bytes, 640000 bits) ----------
proc 0 <-> proc 1 connection bandwidth: 85.6575 Mbit/sec
proc 0 <-> proc 2 connection bandwidth: 85.362 Mbit/sec
proc 0 <-> proc 3 connection bandwidth: 85.2638 Mbit/sec
proc 1 <-> proc 2 connection bandwidth: 85.0122 Mbit/sec
proc 1 <-> proc 3 connection bandwidth: 85.4974 Mbit/sec
proc 2 <-> proc 3 connection bandwidth: 87.4603 Mbit/sec
Results for chunk_size = 1000 double, (8000 bytes, 64000 bits) ----------
proc 0 <-> proc 1 connection bandwidth: 68.7364 Mbit/sec
proc 0 <-> proc 2 connection bandwidth: 67.4536 Mbit/sec
proc 0 <-> proc 3 connection bandwidth: 69.9962 Mbit/sec
proc 1 <-> proc 2 connection bandwidth: 68.7026 Mbit/sec
proc 1 <-> proc 3 connection bandwidth: 68.9142 Mbit/sec
proc 2 <-> proc 3 connection bandwidth: 68.774 Mbit/sec
Results for chunk_size = 100 double, (800 bytes, 6400 bits) ----------
proc 0 <-> proc 1 connection bandwidth: 27.8825 Mbit/sec
proc 0 <-> proc 2 connection bandwidth: 27.8742 Mbit/sec
proc 0 <-> proc 3 connection bandwidth: 28.0218 Mbit/sec
proc 1 <-> proc 2 connection bandwidth: 28.0202 Mbit/sec
proc 1 <-> proc 3 connection bandwidth: 28.0597 Mbit/sec
proc 2 <-> proc 3 connection bandwidth: 28.1122 Mbit/sec
Results for chunk_size = 10 double, (80 bytes, 640 bits) ----------
proc 0 <-> proc 1 connection bandwidth: 7.08554 Mbit/sec
proc 0 <-> proc 2 connection bandwidth: 7.09283 Mbit/sec
proc 0 <-> proc 3 connection bandwidth: 7.11515 Mbit/sec
proc 1 <-> proc 2 connection bandwidth: 7.18078 Mbit/sec
proc 1 <-> proc 3 connection bandwidth: 7.1838 Mbit/sec
proc 2 <-> proc 3 connection bandwidth: 7.19908 Mbit/sec

• Calculo de la latencia

	Long del 
  vector en	Tiempo de 
	bits.		  envio (mu sec)  ancho de banda (Mbit/sec)

	6.4000e+06	7.0878e+04	  90.2961
	6.4000e+05	7.4707e+03	  85.6681
	6.4000e+04	9.2668e+02	  69.0636
	6.4000e+03	2.2941e+02	  27.8971
	6.4000e+02	9.0423e+01		7.0778

La ordenada al origen da la latencia O(150 musec)

-- MarioStorti - 29 Oct 2001