The mpi API on metaserver at Caltech-Dev

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition which fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-Dev

11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:07 PST 11/07/09-00:05:07 GMT 941587522 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP execOverload overloaded exec
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP closeListenSock no cid registered for service 'data'
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP mpi::init unused data port 10021 closed
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP mpi::init port 10021 (jobstate) opened on metaserver as sock3
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP bgLoop Looping process watchlogs started
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP openListenSock port 10019 (operator) opened on metaserver as sock4
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP openListenSock port 10020 (emergency) opened on metaserver as sock5
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP setResourceLimit vmemoryuse=unlimited; datasize=unlimited; core=unlimited; maxproc=65536; descriptors=1024; memorylocked=32768; filesize=unlimited; cputime=unlimited
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP leakLogger inital size of mpi API: 14348 kB
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP bgLoop Looping process etchosts started
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP mpi Trying API
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP mpi API yes
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP mpi Trying LDAS_SYSTEM
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 STARTUP mpi LDAS_SYSTEM yes
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 IDLE exec exec /usr/sbin/lsof -b -p 7623
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 IDLE bgLoop Looping process statpagefile started
11/06/09-16:05:08 PST 11/07/09-00:05:08 GMT 941587523 IDLE bgLoop Looping process killedjobreaper started
11/06/09-16:05:12 PST 11/07/09-00:05:12 GMT 941587527 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.248') (::FTPDIR '') (::HTTPURL 'http://131.215.115.248/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-dev 131.215.115.248') (::LDAS_SYSTEM 'ldas-dev') (::RUNCODE 'LDAS-DEV')
11/06/09-16:05:12 PST 11/07/09-00:05:12 GMT 941587527 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.248') (::FTPDIR '') (::HTTPURL 'http://131.215.115.248/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-dev 131.215.115.248') (::LDAS_SYSTEM 'ldas-dev') (::RUNCODE 'LDAS-DEV')
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 STARTUP mpi::killAllMpirun cleaning up for user ldas
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 IDLE ::exec exec /usr/bin/ssh -n -l ldas beowulf /bin/rm -rf /tmp/lam-ldas@*
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:13 PST 11/07/09-00:05:13 GMT 941587528 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 IDLE ::exec exec /usr/bin/ssh -n beowulf sudo pkill -9 -u search01,search02,search03,search04,search05,search06,search07,search08,search09,search10,search11,search12,search13,search14,search15,search16 wrapperAPI
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:16 PST 11/07/09-00:05:16 GMT 941587531 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 IDLE ::exec exec /usr/bin/ssh -n beowulf sudo pkill -9 -u search01,search02,search03,search04,search05,search06,search07,search08,search09,search10,search11,search12,search13,search14,search15,search16 lamd
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:19 PST 11/07/09-00:05:19 GMT 941587534 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 STARTUP mpi::killAllMpirun ran kill 10 times in 9.042 seconds
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 STARTUP mpi::prestartLamds running lamboot for user search01
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 IDLE ::exec exec scp -B conf.lam search01@metaserver:
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 IDLE ::exec exec scp -B conf.lam search01@metaserver:
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 CXX_THREAD fork_parent@MountPointStatus.cc end
emaros@ligo.caltech.edu 941587537 STARTUP mpi::prestartLamds Subject: LDAS-DEV mpiAPI lamboot error!; Body: lam::boot: lam::pushSchema: search01@metaserver: Permission denied (publickey,gssapi-with-mic,password). lost connection
11/06/09-16:05:22 PST 11/07/09-00:05:22 GMT 941587537 STARTUP mpi::killAllMpirun {mpi::cleanupTmp: ldas@beowulf: ssh: connect to host beowulf port 22: No route to host} {ldas@beowulf:wrapperAPI: ssh: connect to host beowulf port 22: No route to host} {ldas@beowulf:lamd: ssh: connect to host beowulf port 22: No route to host}
11/06/09-16:05:28 PST 11/07/09-00:05:28 GMT 941587543 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
11/06/09-16:05:33 PST 11/07/09-00:05:33 GMT 941587548 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'