Difference: UserGuideEMI2 (14 vs. 15)

Revision 152012-03-22 - AlviseDorigo

Line: 1 to 1
 
META TOPICPARENT name="UserDocumentation"

CREAM User's Guide for EMI-2

Line: 964 to 964
  /bin/sleep 120
Deleted:
<
<		 0
 
Line: 1009 to 1008
  $ glite-es-activity-status -e cream-10.pd.infn.it CR_ES146448265 ***************************************
Changed:
<
<		JobID = CR_ES853695732
>
>		ActivityID = CR_ES853695732
 Status = PROCESSING_RUNNING Attributes = {APP_RUNNING} Timestamp = Tue Mar 20 11:05:28 2012
Line: 1022 to 1021
  $ glite-es-activity-status -e cream-10.pd.infn.it CR_ES146448265 ***************************************
Changed:
<
<		JobID = CR_ES853695732
>
>		ActivityID = CR_ES853695732
 Status = TERMINAL Attributes = {} Timestamp = Tue Mar 20 11:07:28 2012
Line: 1030 to 1029
 
Changed:
<
<		Of course the user can specify more than one activity identifier as argument of status command:
>
>		The user can specify more than one activity identifier as arguments of status command:
 

$ glite-es-activity-status -e cream-10.pd.infn.it CR_ES918217695 CR_ES499424509
*****************************************
Changed:
<
<		JobID = CR_ES499424509
>
>		ActivityID = CR_ES499424509
 Status = TERMINAL Attributes = {} Timestamp = Wed Mar 21 09:45:57 2012 Description = reason=0 ***************************************
Changed:
<
<		JobID = CR_ES918217695
>
>		ActivityID = CR_ES918217695
 Status = TERMINAL Attributes = {} Timestamp = Wed Mar 21 09:45:48 2012
Line: 1049 to 1048
 
Added:
>
>		If the user asks for the status of an acitivity identifier not present on the endpoint, he/she will get an error message like this: 

$ glite-es-activity-status -e cream-05.pd.infn.it CR_ES920948151 FOO
*****************************************
ActivityID  = CR_ES920948151
Status      = TERMINAL
Attributes  = {CLIENT_STAGEOUT_POSSIBLE}
Timestamp   = Thu Mar 22 13:29:11 2012
Description = reason=0
*****************************************
ActivityID  = FOO
Message     = Activity not found!
Timestamp   = Thu Mar 22 14:34:56 2012
Description = N/A
FailCode    = N/A
 

Creating activities on ES based CEs, that need to move Input/Output sandboxes

Automatically stage-in/stage-out performed by the CE

Line: 1079 to 1097
  myjob.sh
Deleted:
<
<		 0
  MY_ENV
Line: 1168 to 1185
  myjob.sh
Deleted:
<
<		 0
  MY_ENV
Line: 1222 to 1237
 

Note the status attribute CLIENT_STAGEIN_POSSIBLE: the CE is saying that it is waiting for the user to send the required ISB by hand (by mean of the usual GridFTP client for example). Also note that the CE is communicating the complete address where to send the ISB (and the protocol supported, gsiftp):

Deleted:
<
<
 
Changed:
<
<		globus-url-copy myjob.sh gsiftp://cream-10.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam004/61/CR_ES613269972/ISB/myjob.sh
>
>		 $ globus-url-copy myjob.sh gsiftp://cream-10.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam004/61/CR_ES613269972/ISB/myjob.sh
 

When the stage-in is finished, the user has to notify the service that the file shipment is complete:

Line: 1291 to 1305
  $ glite-es-activity-create -e cream-05.pd.infn.it ~/JDLs/activity_files_push_pull.adl ***************************************
Changed:
<
<		ActivityID = CR_ES342891609
>
>		ActivityID = CR_ES920948151
 ActivityMgrURI = https://cream-05.pd.infn.it:8443/ce-cream-es/services/ActivityManagementService Status = PREPROCESSING Status Attrs = {CLIENT_STAGEIN_POSSIBLE}
Changed:
<
<		Timestamp = Thu Mar 22 12:05:01 2012
>
>		Timestamp = Thu Mar 22 13:28:32 2012
 Description = ETNSC =
Changed:
<
<		STAGEIN Dir = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/34/CR_ES342891609/ISB}
>
>		STAGEIN Dir = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/92/CR_ES920948151/ISB}
 SESSION Dir = {}
Changed:
<
<		STAGEOUT Dir = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/34/CR_ES342891609/OSB}
>
>		STAGEOUT Dir = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/92/CR_ES920948151/OSB}
 
Changed:
<
<		This time, in addition to the "STAGEIN Dir", the user has to consider the value of "STAGEOUT Dir"; when the activity will be finished, he/she will have to retrieve the two files by mean of the usual GridFTP client:
>
>		This time, in addition to the "STAGEIN Dir", the user has to consider (and remember) the value of "STAGEOUT Dir"; when the activity will be finished (use glite-es-activity-status to check that), he/she will have to retrieve the two files by mean of the usual GridFTP client: 

$ glite-es-activity-status  -e cream-05.pd.infn.it CR_ES920948151
 
Added:
>
>		*************************************** ActivityID = CR_ES920948151 Status = TERMINAL Attributes = {CLIENT_STAGEOUT_POSSIBLE} Timestamp = Thu Mar 22 13:29:11 2012 Description = reason=0

$ globus-url-copy gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/92/CR_ES920948151/OSB/JobOutput.txt JobOutput.txt $ globus-url-copy gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/92/CR_ES920948151/OSB/JobError.txt JobError.txt

When output file retrieve is finished the user should notify the server with the command:

 
Added:
>
>		$ glite-es-notify-service CR_ES920948151:CLIENT-DATAPULL-DONE -e cream-05.pd.infn.it

Now the status's attributes CLIENT_STAGEOUT_POSSIBLE disappears: 


dorigoa@lxgrid05 14:46:46 ~/emi/creamui_emi2>stage/usr/bin/glite-es-activity-status CR_ES920948151 -e cream-05.pd.infn.it
*****************************************
ActivityID  = CR_ES920948151
Status      = TERMINAL
Attributes  = {}
Timestamp   = Thu Mar 22 13:29:11 2012
Description = reason=0

Using an output file for the command glite-es-activity-create

The activity creation command has the possibility to write (just) the activity identifier(s) into an output file. When this file is first created the user has nothing to take care about. If the file already exists, the user has to make sure that the output file has been already used for the same endpoint of the current activity creation. En example is better than any explanation (the special ADL file ~/JDLs/activity_sleep60_multiple.adl cointains 3 descriptions for 3 activities): 

$ glite-es-activity-create -e cream-05.pd.infn.it ~/JDLs/activity_sleep60_multiple.adl -o ids_cream-05 
*****************************************
ActivityID     = CR_ES588248744
ActivityMgrURI = https://cream-05.pd.infn.it:8443/ce-cream-es/services/ActivityManagementService
Status         = PREPROCESSING
Status Attrs   = {}
Timestamp      = Thu Mar 22 14:03:48 2012
Description    = 
ETNSC          = 
STAGEIN Dir    = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/58/CR_ES588248744/ISB}
SESSION Dir    = {}
STAGEOUT Dir   = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/58/CR_ES588248744/OSB}
*****************************************
ActivityID     = CR_ES477186035
ActivityMgrURI = https://cream-05.pd.infn.it:8443/ce-cream-es/services/ActivityManagementService
Status         = PREPROCESSING
Status Attrs   = {}
Timestamp      = Thu Mar 22 14:03:48 2012
Description    = 
ETNSC          = 
STAGEIN Dir    = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/47/CR_ES477186035/ISB}
SESSION Dir    = {}
STAGEOUT Dir   = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/47/CR_ES477186035/OSB}
*****************************************
ActivityID     = CR_ES276464760
ActivityMgrURI = https://cream-05.pd.infn.it:8443/ce-cream-es/services/ActivityManagementService
Status         = PREPROCESSING
Status Attrs   = {}
Timestamp      = Thu Mar 22 14:03:49 2012
Description    = 
ETNSC          = 
STAGEIN Dir    = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/27/CR_ES276464760/ISB}
SESSION Dir    = {}
STAGEOUT Dir   = {gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/27/CR_ES276464760/OSB}

Note the option -o ids_cream-05 that instruct the command to write (only) the activity identifiers on the output file ids_cream-05. Let's take a look at this file: 

$ cat ids_cream-05 
##ACTIVITY_IDS@cream-05.pd.infn.it:8443##
CR_ES588248744
CR_ES477186035
CR_ES276464760

The header indicates the endpoint where the following identifiers has been created. If the user tries to create activites to a different endpoint but wants to write the output identifiers on the file ids_cream-05, he/she will get an error: 


$ glite-es-activity-create -e cream-10.pd.infn.it ~/JDLs/activity_sleep60.adl -o ids_cream-05
*****************************************
ActivityID     = CR_ES341612391
ActivityMgrURI = https://cream-10.pd.infn.it:8443/ce-cream-es/services/ActivityManagementService
Status         = PREPROCESSING
Status Attrs   = {}
Timestamp      = Thu Mar 22 14:08:52 2012
Description    = 
ETNSC          = 
STAGEIN Dir    = {gsiftp://cream-10.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam004/34/CR_ES341612391/ISB}
SESSION Dir    = {}
STAGEOUT Dir   = {gsiftp://cream-10.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam004/34/CR_ES341612391/OSB}
This output file [ids_cream-05] contains activity identifiers created a on different endpoint (cream-05.pd.infn.it:8443). Will not write Activity ID on this file.

Note the error message at the end of the output: This output file [ids_cream-05] contains activity identifiers created a on different endpoint (cream-05.pd.infn.it:8443). Will not write Activity ID on this file.. In fact the endpoint of this command was cream-10.pd.infn.it.

Using an input file for the command glite-es-activity-status

We've seen above that the command glite-es-activity-status accepts one or more activity identifiers as arguments. Alternatively, if the identifiers are many, it could be convenient to load them from an input file, previously created by the glite-es-activity-create command with the -o option. In this case the user must not specify the endpoint where the identifiers have been created, because the endpoint will be extracted from the input file: 


$ glite-es-activity-status -e cream-05.pd.infn.it -i ids_cream-05 
Options --endpoint and --input are mutually exclusive. Stop.

$ glite-es-activity-status -i ids_cream-05
*****************************************
ActivityID  = CR_ES276464760
Status      = TERMINAL
Attributes  = {}
Timestamp   = Thu Mar 22 14:05:15 2012
Description = reason=0
*****************************************
ActivityID  = CR_ES477186035
Status      = TERMINAL
Attributes  = {}
Timestamp   = Thu Mar 22 14:05:17 2012
Description = reason=0
*****************************************
ActivityID  = CR_ES588248744
Status      = TERMINAL
Attributes  = {}
Timestamp   = Thu Mar 22 14:05:05 2012
Description = reason=0

Another possibility is to mix identifiers specified as arguments and an input file: 


$ glite-es-activity-status CR_ES920948151 -i ids_cream-05 
*****************************************
ActivityID  = CR_ES276464760
Status      = TERMINAL
Attributes  = {}
Timestamp   = Thu Mar 22 14:05:15 2012
Description = reason=0
*****************************************
ActivityID  = CR_ES477186035
Status      = TERMINAL
Attributes  = {}
Timestamp   = Thu Mar 22 14:05:17 2012
Description = reason=0
*****************************************
ActivityID  = CR_ES588248744
Status      = TERMINAL
Attributes  = {}
Timestamp   = Thu Mar 22 14:05:05 2012
Description = reason=0
*****************************************
ActivityID  = CR_ES920948151
Status      = TERMINAL
Attributes  = {CLIENT_STAGEOUT_POSSIBLE}
Timestamp   = Thu Mar 22 13:29:11 2012
Description = reason=0

Obtain information about activities

Activity status

The command glite-es-activity-status has already been explained here and here.

Extended activity's information

To obtain extended information about an activity the command glite-es-activity-info can be used with exactly the same syntax and options than glite-es-activity-status : 

$ glite-es-activity-info CR_ES920948151 -e cream-05.pd.infn.it

**** ActivityID = CR_ES920948151

ID=https://cream-05.pd.infn.it:8443/ce-cream-es/services/ActivityManagementService?CR_ES920948151
Name=CustomJob
OtherInfo={}
CreationTime=1332419311 (Thu Mar 22 13:28:31 2012)
Validity=N/A
***** Extension #0:
      LocalID=CR_ES920948151
      Key=ACTIVITY_HISTORY
      Value=ACTIVITY_HISTORY
***** Extension #1:
      LocalID=CR_ES920948151
      Key=STAGE_IN_URI
      Value=gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/92/CR_ES920948151/ISB
***** Extension #2:
      LocalID=CR_ES920948151
      Key=STAGE_OUT_URI
      Value=gsiftp://cream-05.pd.infn.it/var/cream-es_sandbox/dteam/CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL_dteam002/92/CR_ES920948151/OSB
BaseType=
Type=SINGLE
IDFromEndpoint=https://cream-05.pd.infn.it:8443/ce-cream-es/services/ActivityManagementService?CR_ES920948151
LocalIDFromManager=lsf/20120322/620153
JobDescription=emi:adl
State={ACCEPTED, PREPROCESSING, PROCESSING-ACCEPTING, PROCESSING-QUEUED, PROCESSING-RUNNING, TERMINAL}
RestartState={}
ExitCode=1
ComputingManagerExitCode=N/A
Error={}
WaitingPosition=N/A
UserDomain=dteam
Owner=CN_Alvise_Dorigo_L_Padova_OU_Personal_Certificate_O_INFN_C_IT_dteam_Role_NULL_Capability_NULL
LocalOwner=dteam002
RequestedTotalWallTime=N/A
RequestedTotalCPUTime=N/A
RequestedSlots=N/A
RequestedApplicationEnvironment={}
StdIn=JobOutput.txt
StdOut=N/A
StdErr=JobError.txt
LogDir=N/A
ExecutionNode=prod-wn-001.pn.pd.infn.it
Queue=creamtest2
UsedTotalWallTime=N/A
UsedMainMemory=N/A
SubmissionTime=1332419311 (Thu Mar 22 13:28:31 2012)
ComputingManagerSubmissionTime=1332419344 (Thu Mar 22 13:29:04 2012)
StartTime=1332419351 (Thu Mar 22 13:29:11 2012)
ComputingManagerEndTime=1332419351 (Thu Mar 22 13:29:11 2012)
EndTime=1332419351 (Thu Mar 22 13:29:11 2012)
WorkingAreaEraseTime=N/A
ProxyExpirationTime=N/A
SubmissionHost=N/A
SubmissionClientName=N/A
OtherMessages=

Refer to the EMI-ES documentation for a complete explanation of this output. As for the status, also with this command more than one activity identifier can be specified as argument, or an input file, or both.

Obtain the activites created on an endpoint

To get a list of activities the user has created on a certain endpoint, the command glite-es-activity-list can be used: 


$ glite-es-activity-list cream-05.pd.infn.it
CR_ES075634021
CR_ES117045296
CR_ES225992668
CR_ES276464760
CR_ES330911211
CR_ES342891609
CR_ES477186035
CR_ES556329337
CR_ES588248744
CR_ES800305261
CR_ES832095061
CR_ES920948151

A certain user can see ONLY his/her activity identifiers (those ones created with his/her proxy certificate).

glite-es-activity-list can redirect its output into a file; if this file already exists, it must be of the same type of the glite-es-activity-create 's one, i.e. it must contain the usual header line: 


$ cat ids_cream-05 
##ACTIVITY_IDS@cream-05.pd.infn.it:8443##
CR_ES588248744
CR_ES477186035
CR_ES276464760

The option to use is again -o : 


$ glite-es-activity-list cream-05.pd.infn.it -o cream-05_activity_list
CR_ES075634021
CR_ES117045296
CR_ES225992668
CR_ES276464760
CR_ES330911211
CR_ES342891609
CR_ES477186035
CR_ES556329337
CR_ES588248744
CR_ES800305261
CR_ES832095061
CR_ES920948151

$ cat cream-05_activity_list
##ACTIVITY_IDS@cream-05.pd.infn.it:8443##
CR_ES075634021
CR_ES117045296
CR_ES225992668
CR_ES276464760
CR_ES330911211
CR_ES342891609
CR_ES477186035
CR_ES556329337
CR_ES588248744
CR_ES800305261
CR_ES832095061
CR_ES920948151

Once created (or updated), this file can be used to feed glite-es-activity-status or glite-es-activity-info . As for the activity creation, if the output file already exists, it must contain activity identifiers created on the same endpoint where the user is asking the list currently: 


$ head -1 cream-10_activity_list 
##ACTIVITY_IDS@cream-10.pd.infn.it:8443##

$ glite-es-activity-list cream-05.pd.infn.it -o cream-10_activity_list
CR_ES075634021
CR_ES117045296
CR_ES225992668
CR_ES276464760
CR_ES330911211
CR_ES342891609
CR_ES477186035
CR_ES556329337
CR_ES588248744
CR_ES800305261
CR_ES832095061
CR_ES920948151
This output file [cream-10_activity_list] contains activity identifiers created a on different endpoint (cream-10.pd.infn.it:8443). Will not write Activity ID on this file.
 
View topic | History: r25 < r24 < r23 < r22 | More topic actions...
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback