[egiorgio@ui2 ~]$ lfc-mkdir /grid/gridit/gtw/$USER

We are ready to store a file on the grid. Create locally a simple text file

• Virtual Organization membership : this tutorial assumes VO gridit membership, but it can be smoothly used with any another VO of your choice, just replacing in the examples the VO name.

Connecting to the service https://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server

================== glite-wms-job-submit Success ==================

The job has been successfully submitted to the WMProxy Your job identifier is:

https://wms005.cnaf.infn.it:9000/a-UJrAT2A3zSkScTngAc6Q

The job identifier has been saved in the following file: /home/egiorgio/jobid.txt

======================================================================

=================== glite-wms-job-status Success ================= BOOKKEEPING INFORMATION:

Status info for the Job : https://wms005.cnaf.infn.it:9000/a-UJrAT2A3zSkScTngAc6Q Current Status: Running Status Reason: unavailable Destination: ce-1.le.infn.it:8443/cream-lsf-gridit Submitted: Tue Jan 14 11:34:42 2014 CET ====================================================================== [egiorgio@ui2 ~]$ glite-wms-job-status -i jobid.txt

=================== glite-wms-job-status Success ================= BOOKKEEPING INFORMATION:

Status info for the Job : https://wms005.cnaf.infn.it:9000/a-UJrAT2A3zSkScTngAc6Q Current Status: Done(Success) Logged Reason(s): - job completed - Job Terminated Successfully Exit code: 0 Status Reason: Job Terminated Successfully Destination: ce-1.le.infn.it:8443/cream-lsf-gridit Submitted: Tue Jan 14 11:34:42 2014 CET ======================================================================

Once that you get Job Done message, you can download the job output with glite-wms-job-output command. You can either provide as input the job id, as well as the text file where you could have saved the job id. Notice that, according to the UI settings, you might have to specify the downloading directory, via the --dir switch. In the given example, we are storing the output in the current directory (.).

[egiorgio@ui2 ~]$ glite-wms-job-output -i jobid.txt --dir .

Connecting to the service https://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server

It's better to check on the UI whether a LFC server is set or not.

[egiorgio@ui2 ~]$ echo $LFC_HOST 
lfcserver.cnaf.infn.it
[egiorgio@ui2 ~]$ 

Should the variable $LFC_HOST be unset, just set it with export LFC_HOST=. You can retrieve a list of LFC servers suitable for your VO using lcg-infosites --vo yourVO lfc

• Browse storage and computing resources for a given Virtual Organization
• how to submit a job to the grid computing resources

[egiorgio@ui2 ~]$ voms-proxy-init --voms gridit 
Unable to find user certificate or key: /home/egiorgio/.globus/usercred.p12
[egiorgio@ui2 ~]$ mkdir .globus
[change machine...]
emidio@mybox: $ scp cert_INFN_2014.p12 egiorgio@ui2.grid.unipg.it:.globus/usercred.p12
cert_INFN_2014.p12   
emidio@mybox: ~ $ ssh egiorgio@ui2.grid.unipg.it
Last login: Thu Jan  9 09:21:08 2014

The Information System

You can see the computing and storage resources available for your VO with the command lcg-infosites  --vo yourVO (ce|se)

Tip A computing resource is shortly said ce (computing element), while a storage resource (storage element) is said se.

[egiorgio@ui2 ~]$ lcg-infosites --vo gridit ce
#   CPU	   Free	Total Jobs	Running	Waiting	ComputingElement
----------------------------------------------------------------
      8	      8	         0	      0	      0	atlasce2.lnf.infn.it:8443/cream-pbs-grid
   1232	      0	         0	      0	      0	atlasce3.lnf.infn.it:8443/cream-pbs-grid
    680	    156	         1	      1	      0	ce-01.roma3.infn.it:8443/cream-pbs-fastgrid
    680	    156	        22	     22	      0	ce-01.roma3.infn.it:8443/cream-pbs-grid
     40	     39	         0	      0	      0	ce-1.le.infn.it:8443/cream-lsf-gridit
[..cut]
[egiorgio@ui2 ~]$ lcg-infosites --vo gridit se
 Avail Space(kB)  Used Space(kB)  Type 	SE
------------------------------------------
       428613066        99825647  SRM  	aliserv6.ct.infn.it
      3654311847      2067559635  SRM  	atlasse.lnf.infn.it
        78100676       221899323  SRM  	darkstorm.cnaf.infn.it
        79062315       220937684  SRM  	darkstorm.cnaf.infn.it
         1402415          597584  SRM  	egee013.cnaf.infn.it
         1999991               8  SRM  	egee013.cnaf.infn.it
[..cut]

Optional lcg-infosites offers more options than the basic listed above. You can check them reading the man page.

Connecting to the service https://gridrb.fe.infn.it:7443/glite_wms_wmproxy_server

Grid Usage in a glance

echo "Hello World - from $HOSTNAME"

The submission command forward the execution request to a service, the WMS, which selects a resource for the actual execution. The output of the submission command contains a job identifier, in this example https://prod-wms-01.ct.infn.it:9000/ROeQZNBlEnG535cX4ATFvQ. This is needed for the monitoring of job status and the output retrieve. Depending on the status of resources, the job execution could take some time. Eventually, the job status command (glite-wms-job-status <job-id>) returns with the job done message, which means you can download the output with glite-wms-job-output <job-id>

emidio@emi-tutor: ~ $ glite-wms-job-status https://prod-wms-01.ct.infn.it:9000/ROeQZNBlEnG535cX4ATFvQ

======================= glite-wms-job-status Success =====================
BOOKKEEPING INFORMATION:

Status info for the Job : https://prod-wms-01.ct.infn.it:9000/ROeQZNBlEnG535cX4ATFvQ
Current Status:     Done(Success)
Logged Reason(s):
    - job completed
    - Job Terminated Successfully
Exit code:          0
Status Reason:      Job Terminated Successfully
Destination:        grid0.fe.infn.it:8443/cream-pbs-grid
Submitted:          Tue Jul 16 16:18:07 2013 CEST
==========================================================================
emidio@emi-tutor: ~ $ glite-wms-job-output https://prod-wms-01.ct.infn.it:9000/ROeQZNBlEnG535cX4ATFvQ

Connecting to the service https://prod-wms-01.ct.infn.it:7443/glite_wms_wmproxy_server

================================================================================

			JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://prod-wms-01.ct.infn.it:9000/ROeQZNBlEnG535cX4ATFvQ
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/emidio_ROeQZNBlEnG535cX4ATFvQ

================================================================================

emidio@emi-tutor: ~ $ cat /tmp/jobOutput/emidio_ROeQZNBlEnG535cX4ATFvQ/stdout.txt 
Hello World - from grid6.fe.infn.it
emidio@emi-tutor: ~ $ 

Data Management

Create a simple text file

emidio@emi-tutor: ~ $ echo "My first grid file" > file.txt
emidio@emi-tutor: ~ $ cat file.txt 
My first grid file
emidio@emi-tutor: ~ $ 

to upload it on a grid storage element, just run the command lcg-cr <path-to-local-file>

emidio@emi-tutor: ~ $ lcg-cr file.txt 
guid:cc002f0c-8615-4c38-8bf7-581d438c7fa6

emidio@emi-tutor: ~ $ lcg-cp guid:cc002f0c-8615-4c38-8bf7-581d438c7fa6 grid-file.txt
emidio@emi-tutor: ~ $ cat grid-file.txt 
My first grid file

Once the file is no more needed, we can delete it from the storage element : lcg-del <grid-identifier> Since we are using the GUID for the file identification, we need to add -a (all) option, to enforce our intention to delete all the copies of the file.

emidio@emi-tutor: ~ $ lcg-del -a guid:cc002f0c-8615-4c38-8bf7-581d438c7fa6
emidio@emi-tutor: ~ $

=glite-wms-job-submit -a <jobname.jdl>

jobname.jdl is a file which describes the computational task you want to execute. For instance

emidio@emi-tutor: ~ $ cat hello.jdl

# the name of the file which will be actually executed
Executable = "hello.sh" ; 

# the name of the file where standard output stream will be redirected
StdOutput = "stdout.txt";

# the name of the file where standard error stream will be redirected
StdError = "stderr.txt";

# list of local files which will be transferred to 
# the remote resource before the execution
InputSandbox = {"hello.sh"};

# list of remote files which will be transferred from the resource 
# after the execution
OutputSandbox = {"stdout.txt", "stderr.txt"};

In this minimal example, a Bash script, hello.sh will be executed. The Bash script is transferred to the remote resource, executed, and the standard output/error are streamed on the files stdout.txt and stderr.txt, which are included in the job Output Sandbox for the successive retrieval. The file has to have mandatory .jdl extension. JDL. which stands for job description language, is a powerful, descriptive language, that allows to specify many tiny details of the task. See the full guide for more details. The -a performs an automatic delegation of your proxy to the remote resource.

emidio@emi-tutor: ~ $ glite-wms-job-submit -a hello.jdl 

Connecting to the service https://prod-wms-01.ct.infn.it:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://prod-wms-01.ct.infn.it:9000/ROeQZNBlEnG535cX4ATFvQ

==========================================================================

The submission command returns a job identifier, in this example https://prod-wms-01.ct.infn.it:9000/ROeQZNBlEnG535cX4ATFvQ. This is needed for the monitoring of job status and the output retrieve

Job Submission

A quick introduction to Grid Usage

About

• how to submit a job to the grid resources
• how to store a file on a grid storage element, and retrieve it back

Goals

• Understand basics of job submission and monitoring.
• Understand grid data management fundamentals.

Prerequisites

• Access to an User Interface
• A personal X509 certificate
• Virtual Organization membership

Contents

Authentication

Once you logged to your account on a User Interface, as first thing you have to create a proxy. This will authenticate yourself towards grid services. To create your proxy, you need to have your certificate in the .globus directory, and you need to be member of a Virtual Organisation (VO). If the VO you belong is configured on the User Interface, the proxy certificate can be created, and will be used by other grid client in order to authenticate your request to the grid services. You are asked for a password, it is the password you created when you exported the certificate from your browser.

emidio@emi-tutor: ~ $ ls -l .globus
total 4
-rw------- 1 emidio emidio 3852 Jul 16 11:13 usercred.p12
emidio@emi-tutor: ~ $ voms-proxy-init --voms gridit
Enter GRID pass phrase for this identity:
Contacting voms.cnaf.infn.it:15008 [/C=IT/O=INFN/OU=Host/L=CNAF/CN=voms.cnaf.infn.it] "gridit"...
Remote VOMS server contacted succesfully.

Created proxy in /tmp/x509up_u500.

Your proxy is valid until Wed Jul 17 01:08:54 CEST 2013

Further Material

Copyright © 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback