The CREAM user's guide is available at:
https://edms.cern.ch/document/595770
CREAM User's Guide
1 CREAM Command Line Interface Guide
This section briefly explains the sequence of operations to be performed by a user to submit and then manage jobs on CREAM based CEs, referring to the C++ Command Line Interface (CLI).
1.1 Before starting: get your user proxy
Before using any of the CREAM client commands, it is necessary to have a valid proxy credential available on the client machine. You can create it using the
voms-proxy-init
command. If you already have a valid proxy available on your machine just make the
X509_USER_PROXY
environment variable point to it.
In order to get a proxy certificate issued by VOMS, you should have in the directory
/etc/vomses
the proper VOMS file containing a line as follows:
"EGEE" "kuiken.nikhef.nl" "15001" "/O=dutchgrid/O=hosts/OU=nikhef.nl/CN=kuiken.nikhef.nl" "EGEE" "22"
or the corresponding line for your VO. You also need to install the VO related .lsc files in the
/etc/grid-security/vomsdir/<VO>
directory. In a standard EMI UI installation, these settings should be already there.
Make moreover sure you have in the directory
$HOME/.globus
your certificate/key pair, i.e. the following files:
usercert.pem
userkey.pem
Note that file permissions are important: the two files must have respectively 0600 and 0400 permissions.
Then you can issue the VOMS client command (you will be prompted for the pass-phrase):
$ voms-proxy-init -voms dteam
Enter GRID pass phrase:
Your identity: /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto
Creating temporary proxy ............................................................................................................................................... Done
Contacting voms2.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr] "dteam" Done
Creating proxy .............................. Done
Your proxy is valid until Sat Apr 30 05:05:49 2011
$ voms-proxy-info -all
subject : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto/CN=proxy
issuer : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto
identity : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto
type : proxy
strength : 1024 bits
path : /tmp/x509up_u500
timeleft : 11:59:55
key usage : Digital Signature, Key Encipherment, Data Encipherment
=== VO dteam extension information ===
VO : dteam
subject : /C=IT/O=INFN/OU=Personal Certificate/L=Padova/CN=Massimo Sgaravatto
issuer : /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr
attribute : /dteam/Role=NULL/Capability=NULL
attribute : /dteam/italy/Role=NULL/Capability=NULL
attribute : /dteam/italy/INFN-PADOVA/Role=NULL/Capability=NULL
timeleft : 11:59:55
uri : voms2.hellasgrid.gr:15004
1.2 CREAM CLI commands
The most relevant commands to interact with CREAM based CEs are:
- glite-ce-job-submit <jdl_file>
- glite-ce-delegate-proxy <delegation_Id>
- glite-ce-job-status <job_Ids>
- glite-ce-job-list <host[:port]>
- glite-ce-job-cancel <job_Ids>
- glite-ce-job-suspend <job_Ids>
- glite-ce-job-resume <job_Ids>
- glite-ce-job-output <job_Ids>
- glite-ce-job-purge <job_Ids>
- glite-ce-proxy-renew <delegation_Ids>
- glite-ce-service-info <host[:port]>
- glite-ce-get-cemon-url <host[:port]>
- glite-ce-enable-submission <host[:port]>
- glite-ce-disable-submission <host[:port]>
- glite-ce-allowed-submission <host[:port]>
Man pages are available for all the CREAM client commands. You can also access information about the usage of each command by issuing:
$ <command> --help
glite-ce-job-submit
submits a job to a CREAM based CE. It requires a JDL file as input and returns a CREAM job identifier.
glite-ce-delegate-proxy
allows the user to delegate her proxy credential to the CREAM service. This delegated credential can then be used for job submissions.
glite-ce-job-status
displays information (in particular the states) of jobs previously submitted to CREAM based CEs.
glite-ce-job-list
lists the identifiers of jobs submitted to a CREAM based CE by the user issuing the command.
glite-ce-job-cancel
cancels one or more jobs previously submitted to CREAM based CEs.
glite-ce-job-suspend
suspends the execution of one or more jobs previously submitted to CREAM based CEs.
glite-ce-job-resume
resumes the execution of one or more jobs which have been previously suspended.
glite-ce-job-output
retrieves the output sandbox files of one or more jobs previously submitted to CREAM based CEs.
glite-ce-job-purge
clears one or more jobs from CREAM based CEs. After this operation the purged jobs can’t be managed anymore.
glite-ce-proxy-renew
renews delegations, and therefore refreshes the proxy of the jobs submitted to CREAM based CEs using the considered delegations.
glite-ce-service-info
returns information about the CREAM service (version, status, etc.).
glite-ce-get-cemon-url
returns the end-point of the CEMon service coupled with the considered CREAM CE.
glite-ce-enable-submission
(re-)enables job submissions on the specified CREAM CE.
glite-ce-disable-submission
disables job submissions on the specified CREAM CE.
glite-ce-allowed-submission
checks if jobs submissions on the specified CREAM CE are allowed or have been disabled.
All these commands are described in the following sections.
1.3 Submitting jobs to CREAM based CEs
To submit jobs to CREAM based CEs, the command
glite-ce-job-submit
must be used. The glite-ce-job-submit command requires as input a job description file, which describes the job characteristics and requirements through the JDL (Job Description Language).
A typical example of a JDL job description file is:
[
Type = "Job";
JobType = "Normal";
Executable = "myexe";
StdInput = "myinput.txt";
StdOutput = "message.txt";
StdError = "error.txt";
InputSandbox = {"/users/seredova/example/myinput.txt",
"/users/seredova/example/myexe"};
OutputSandbox = {"message.txt", "error.txt"};
OutputSandboxBaseDestUri = "gsiftp://se.pd.infn.it/data/seredova";
]
Such a JDL would make the
myexe
executable be transferred on the remote CREAM CE and be run taking the
myinput.txt
file (also copied from the client node) as input. The standard streams of the job are redirected to files
message.txt
and
error.txt
, and when job completes its execution they are automatically
uploaded on
gsiftp://se.pd.infn.it/data/seredova
.
A detailed description of the available JDL attributes and of the rules for building correct JDL files is provided at
http://wiki.italiangrid.org/twiki/bin/view/CREAM/JdlGuide.
Each job submitted to a CREAM based CE is given the delegated credentials of the user who submitted it. These credentials can then be used when operations requiring security support has to be performed by the job.
There are two possible options to deal with proxy delegation:
- asking the “automatic” delegation of the credentials during the submission operation;
- explicitly delegating credentials, and then asking to rely on these previously delegated credentials on the actual submission operations.
It is highly suggested to rely on this latter mechanism, using the same delegated proxy for multiple job submissions, instead of delegating each time a proxy. Delegating a proxy, in fact, is an operation that can require a non negligible time.
The command
glite-ce-delegate-proxy
is the command to be used to explicitly delegate the user credentials to a CREAM CE.
The following shows an example of job submission, performed explicitly delegating credentials.
So first of all the credentials are delegated to a CREAM based CE (whose endpoint is specified with the option
–endpoint (-e)
:
> glite-ce-delegate-proxy -e cream-ce-01.pd.infn.it mydelid
2006-02-26 15:03:37,286 NOTICE - Proxy with delegation id [mydelid] successfully
delegated to endpoint [https://cream-ce-01.pd.infn.it:8443//ce-cream/services/CREAMDelegation]
The identifier of the delegation is then specified with the
–delegationId (-D)
option in the job submit operation:
> glite-ce-job-submit -D mydelid -r cream-ce-01.pd.infn.it:8443/cream-lsf-grid02 myjob1.jdl
The option
-r (–resource)
has been used to specify the identifier of the CREAM CE where the job has to be submitted to.
myjob1.jdl
is the JDL file describing the job to be submitted.
The command returns the CREAM job identifier associated with this job (e.g.
https://cream-ce-01.pd.infn.it:8443/CREAM116j9vgnf
) which identifies it in clear and unique way all over the Grid system scope.
1.4 Monitoring jobs
2 Man pages for CREAM Command Line Interface
3 Forward of requirements to the batch system
The CREAM CE allows to forward, via tha BLAH component, requirements to the batch system.
For this purpose the JDL =CERequirements- attribute, described at
http://wiki.italiangrid.org/twiki/bin/view/CREAM/JdlGuide#3_27_CERequirements, can be used.
For direct submissions to the CREAM CE (e.g. jobs submitted to the CREAM CE using the CREAM CLI
glite-ce-job-submit
command) the CeRequirements attribute is supposed to be filled by the end-user.
For jobs submitted to the CREAM CE via the WMS, the CeRequirements attribute is instead filled by the WMS, considering the JDL
Requirements
expression and the value of the
CeForwardParameters
attribute in the WMS configuration file.
For example, if in the user JDL there is :
Requirements= "other.GlueHostMainMemoryRAMSize > 100 && other.GlueCEImplementationName==\"CREAM\"";
and if the WMS configuration file there is:
CeForwardParameters = {"GlueHostMainMemoryVirtualSize","GlueHostMainMemoryRAMSize", "GlueCEPolicyMaxCPUTime"};
in the JDL sent by the WMS to CREAM there will be:
CeRequirements= "other.GlueHostMainMemoryRAMSize > 100";
The CERequirements expression received by CREAM is then forwarded to BLAH. Basically BLAH manages the CERequirements expression setting some environment variables, which are available and can be properly used by the
/usr/bin/xxx_local_submit_attributes.sh
script (e.g.
/usr/bin/pbs_local_submit_attributes.sh
for PBS/Torque,
/usr/bin/lsf_local_submit_attributes.sh
for LSF). This script must be properly created by the site admin.
For example, considering the following CeRequirements expression:
CeRequirements="other.GlueHostMainMemoryRAMSize > 100 && other.GlueCEStateWaitingJobs <10 && other.GlueCEImplementationName==\"CREAM\" && other.GlueHostProcessorClockSpeed >= 2800 && (Member(\"FDTD\", other.GlueHostApplicationSoftwareRuntimeEnvironment))";
the following settings will be available in $GLITE_LOCATION/bin/xxx_local_submit_attributes.sh:
GlueHostMainMemoryRAMSize_Min='100'
GlueCEStateWaitingJobs_Max='10'
GlueCEImplementationName='CREAM'
GlueHostProcessorClockSpeed_Min='2800'
GlueHostApplicationSoftwareRuntimeEnvironment='"FDTD"'
What is printed by the
/usr/bin/xxx_local_submit_attributes.sh
script is automatically added to the submit command file.
For example if the JDL Cerequirements expression is:
CeRequirements = "(Member(\"FDTD\", other.GlueHostApplicationSoftwareRuntimeEnvironment))";
and the
/usr/bin/pbs_local_submit_attributes.sh
is:
#!/bin/sh
if [ "$other.GlueHostApplicationSoftwareRuntimeEnvironment" == "FDTD" ]; then
echo "#PBS -l software=FDTD"
fi
then the PBS submit file that will be used will include:
...
...
# PBS directives:
#PBS -S /bin/bash
#PBS -o /dev/null
#PBS -e /dev/null
#PBS -l software=FDTD
....
....
where the line:
#PBS -l software=FDTD
is set via the
/usr/bin/pbs_local_submit_attributes.sh
script.
Please note that there are no differences if in CeRequirements expresssion there is e.g.
CeRequirements = other.xyz==\"ABC\"
or:
CeRequirements = "xyz==\"ABC\"";
In both cases in
/usr/bin/xxx_local_submit_attributes.sh
the variable
xyz
will be set.
As shown above, having
x>a
or
x>=a
doesn't make any difference in the setting of the environment variable
x
in the
/usr/bin/xxx_local_submit_attributes.sh
script. It will be in both cases:
x_Min='a'
--
MassimoSgaravatto - 2011-04-07