Tags:
, view all tags

WMS Test Plan

YAIM-WMS Configuration Testing

WMS Job Submission/GetOutput Testing

Registers the job into the wmproxy, delivers it to the WM that submits it to the grid. Test job submission with the following type of jobs/jdl:

Normal Job

  • to JC
    [
    Type = "Job";
    JobType = "normal";
    InputSandbox = { "file://$HOME/test.sh"};
    VirtualOrganisation = "dteam";
    Executable="test.sh";
    Arguments="Hello ";
    Requirements = ( !RegExp("/cream-",other.GlueCEUniqueID));
    Rank = 0;
    fuzzyrank = true;
    StdOutput="message.txt";
    StdError="err.log";
    OutputSandbox={"message.txt","err.log",".BrokerInfo"};
    RetryCount = 0;
    usertags = [ jdl = "normal job" ];
    ShallowRetryCount = 3;
    ]
  • to ICE
    [
    Type = "Job";
    JobType = "normal";
    InputSandbox = { "file:///home/emolinari/test.sh"};
    VirtualOrganisation = "dteam";
    Executable="test.sh";
    Arguments="Hello ";
    Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
    Rank = 0;
    fuzzyrank = true;
    StdOutput="message.txt";
    StdError="err.log";
    OutputSandbox={"message.txt","err.log",".BrokerInfo"};
    usertags = [ jdl = "normal job to ICE" ];
    RetryCount = 0;
    ShallowRetryCount = 3;
    ]

Perusal job

Job perusal is the ability to view output from a job while it is running. In order to use this feature the JDL attributes PerusalFileEnable and PerusalTimeInterval must be declared when the job is submitted for example:
PerusalFileEnable = true;
PerusalTimeInterval = 120;
This makes the WN upload, at regular time intervals (PerusalTimeInterval), a copy of the output files specified using the glite-wms-job-perusal --set command to the WMS machine.
  • Following an example on how to test a perusal job:
    • Given the jdl file:
      [
      Executable = "perusal.sh" ;
      InputSandbox = {"file://$HOME/perusal.sh"};
      Stdoutput = "perusal.out" ;
      StdError =  "perusal.err" ;
      OutputSandbox ={ "perusal.out" ,"perusal.err"} ;
      usertags = [ jdl = "perusal" ];
      PerusalFileEnable = true;
      ]
      where perusal.sh is
      #!/bin/bash
      
      echo "Hi all, this is a perusal test program..."
      
      for (( i=0 ; $i < 50 ; i = $(($i + 1)) )); do
      
              echo "Step number $i"
              date
              sleep 10
      
      done
      
      echo ""
      echo "Bye"
    • Submit the job with
       glite-wms-job-submit --config glite_wms.conf -a perusal.jdl
    • Given the jobid of the previously submitted set the files that are to be perused with
      glite-wms-job-perusal --set -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg
    • When the job starts running on the WN get the files as here:
      glite-wms-job-perusal --get -f perusal.out https://devel17.cnaf.infn.it:9000/HBPb0WwhMfhKbjdUkrasWg

DAG job

to JC and IC as above specifying Requirements = ( RegExp("/cream-",other.GlueCEUniqueID)); or Requirements = ( RegExp("/cream-",other.GlueCEUniqueID)); respectively in the job jdl
  • DAG job jdls samples:
    [
    Type = "dag";
    VirtualOrganisation = "dteam";
    Max_nodes_running = 10;
    InputSandbox = "test.sh";
    FuzzyRank = true;
    Nodes = [
    nodeA = [
    file= "test_dag.jdl";
    ];
    nodeB = [
    file= "test_dag.jdl";
    ];
    nodeC = [
    file= "test_dag.jdl";
    ];
    nodeD = [
    file= "test_dag.jdl";
    ];
    nodeE = [
    file= "test_dag.jdl";
    ];
    nodeF= [
    file= "test_dag.jdl";
    ];
    ];
    Dependencies = {
    {{nodeA, nodeB}, nodeC},{nodeD,nodeE.nodeF}
    }
    ]
    [
        type = "dag";
        DefaultNodeShallowRetryCount = 3;
            nodes = [
          nodeA = [
            node_type = "edg-jdl";
            file ="myjob.jdl" ;
          ];
            nodeB = [
                                    node_type = "edg-jdl";
            file ="myjob.jdl" ;
           ];
             nodeC = [
             node_type = "edg-jdl";
             file ="myjob.jdl" ;
           ];
                            dependencies = {
            { nodeA, nodeB },
                                    { nodeA, nodeC }
           }
         ];
      ]

Parametric Job

Submit it to ICE and JC. Following a sample jdl:
// Test parametric job.
// The standar Output and Error are parametrizied such as the
// enviroment variable: MYPATH_PARAM_.
 [
  JobType = "parametric";
  Executable = "/usr/bin/env";
  Environment = {"MYPATH_PARAM_=$PATH:/bin:/usr/bin:$HOME"};
  StdOutput = "echo_PARAM_.out";
  StdError = "echo_PARAM_.err";
  OutputSandbox = {"echo_PARAM_.out","echo_PARAM_.err"};
  Requirements = ( RegExp("/cream-",other.GlueCEUniqueID));
  Parameters =  5;
        usertags = [ jdl = "parametric cream" ];
 ]

Collection Job

WMS Job shallow and deep re-submission

  • For the shallow resubmission add the following attributes to your favourite jdl file:
    Prologue = "/bin/false"; ShallowRetryCount = 3;
  • For the deep resubmission add the following attributes:
     Prologue = "/bin/false"; RetryCount = 3;

WMS Job List-match Testing

With data

  • Set these variables:
      export LCG_GFAL_INFOSYS=<BDII set in the wms conf file> (e.g. cert-bdii-04.cnaf.infn.it:2170)
      export LFC_HOST=<lfc host name> (e.g. lfcserver.cnaf.infn.it)
      export LFC_HOME=<lfc home directory> (e.g. lfcserver.cnaf.infn.it:/grid/infngrid)
  • Register a file to a SE using lcg-cr command:
      lcg-cr --vo <VO name> -d <SE host> -l lfn:<logical file name> file:<local file path>
  • Using the lcg-rep you can create some replicas:
      lcg-rep  --vo <VO name> -d <SE host> lfn:<logical file name>
  • lcg-lr is useful to see the list of replicas:
      lcg-lr --vo <VO name>  lfn:<logical file name>
  • Submit a list-match command with a jdl likes this one (as InputData put the lfn(s) registered before):
    ###########################################
    #      JDL with Data Requirements         #
    ###########################################
    
    Executable = "calc-pi.sh";
    Arguments = "1000";
    StdOutput = "std.out";
    StdError = "std.err";
    Prologue = "prologue.sh";
    InputSandbox = {"calc-pi.sh", "fileA", "fileB","prologue.sh"};
    OutputSandbox = {"std.out", "std.err","out-PI.txt","out-e.txt"};
    Requirements = true;
    
    DataRequirements = {
    [
    DataCatalogType = "DLI";
    DataCatalog = "http://lfcserver.cnaf.infn.it:8085";
    InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"};
    ]
    };
    DataAccessProtocol = "gsiftp";
  • You should obtain an output like this one:
    [ale@cream-15 DataReq]$ glite-wms-job-list-match -a --rank --config ~/UI/etc/wmp_wms007.conf data-req.jdl
    Connecting to the service https://wms007.cnaf.infn.it:7443/glite_wms_wmproxy_server
    
    ==========================================================================
    
             COMPUTING ELEMENT IDs LIST
     The following CE(s) matching your job requirements have been found:
    
      *CEId*                                                   *Rank*
    
     - cert-15.pd.infn.it:8443/cream-lsf-cert                           0
     - gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert           0
     - prod-ce-01.pd.infn.it:8443/cream-lsf-cert                        0
     - prod-ce-02.pd.infn.it:2119/jobmanager-lcglsf-cert                0
     - t2-ce-01.to.infn.it:8443/cream-pbs-cert                          0
     - t2-ce-01.to.infn.it:8443/cream-pbs-short                         0
     - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-cert                  0
     - t2-ce-02.to.infn.it:2119/jobmanager-lcgpbs-short                 0
     - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cert               0
     - test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-parallel           0
     - devce.cnaf.infn.it:8443/cream-pbs-cert                           -700634
    
    ==========================================================================
  • Finally you can check if these CEs are really closed to the SEs where you replicate your file(s)
    • First list the SEs:
      [ale@cream-15 DataReq]$ lcg-lr --vo dteam  lfn:/grid/infngrid/cesini/PI_1M.txt
      
      srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/infngrid/generated/2009-11-24/file0b6a613c-e1f1-4e53-a328-2152e44a8576
      srm://gridit-se-01.cnaf.infn.it/dpm/cnaf.infn.it/home/infngrid/generated/2009-11-24/fileb9f3b972-806c-48a6-92fc-a9677901403a
      srm://prod-se-01.pd.infn.it/infngrid/generated/2009-11-24/file10b5fe11-ad95-41e6-a3a3-14bc5cb9694b
      srm://t2-se-00.to.infn.it/dpm/to.infn.it/home/infngrid/generated/2009-11-24/file1bb28727-1356-4fb1-bb73-5362f93e766f
    • then for each SEs in the list use this command to find the "Close" CEs:
      ldapsearch -x -h <BDII set in the WMS conf file> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(GlueCESEBindGroupSEUniqueID=<SE HOST>))" | grep ^GlueCESEBindGroupCEUniqueID
      
      GlueCESEBindGroupCEUniqueID: devce.cnaf.infn.it:8443/cream-pbs-cert
      GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
      GlueCESEBindGroupCEUniqueID: cert-08.cnaf.infn.it:8443/cream-lsf-pps
      GlueCESEBindGroupCEUniqueID: cert-05.cnaf.infn.it:8443/cream-lsf-pps
      GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-par
      GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
      GlueCESEBindGroupCEUniqueID: cert-07.cnaf.infn.it:8443/cream-lsf-pps
      GlueCESEBindGroupCEUniqueID: cert-09.cnaf.infn.it:8443/cream-lsf-pps
      GlueCESEBindGroupCEUniqueID: gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs
      GlueCESEBindGroupCEUniqueID: cert-06.cnaf.infn.it:8443/cream-lsf-pps
      GlueCESEBindGroupCEUniqueID: cert-04.cnaf.infn.it:8443/cream-lsf-pps
      GlueCESEBindGroupCEUniqueID: test7200a.cnaf.infn.it:2119/jobmanager-lcgpbs-cer
      GlueCESEBindGroupCEUniqueID: cert-13.cnaf.infn.it:8443/cream-lsf-pps
    • You can find more CEs because you should also filter using the "vo" attribute.

Others

BrokerInfo

  • Use a jdl like this one to verify its creation:
    ###########################################
    #    JDL to test Brokerinfo Creation      #
    ###########################################
    
    Executable = "/bin/ls";
    Arguments = "-la";
    StdOutput = "std.out";
    StdError = "std.err";
    FuzzyRank = true;
    InputSandbox = {"calc-pi.sh", "fileA", "fileB"};
    OutputSandbox = {"std.out", "std.err",".BrokerInfo"};
    
    Requirements = (other.GlueCEInfoHostName != "spacin-ce1.dma.unina.it") && !regexp("8443/cream", other.GlueCEUniqueID) ;
    
    DataRequirements = {
    [
    DataCatalogType = "DLI";
    DataCatalog = "http://lfcserver.cnaf.infn.it:8085";
    InputData = {"lfn:/grid/infngrid/cesini/PI_1M.txt","lfn:/grid/infngrid/cesini/e-2M.txt"};
    ]
    };
    
    DataAccessProtocol = "gsiftp";
    
  • After the retrieving of the output files you should see these files:
    [ale@cream-15 DataReq]$ ls -la /tmp/ale_5XbbEhIAtjO5uC5wRyCmyA
    total 1264
    drwxr-xr-x   2 ale  ale     4096 Mar  4 10:31 .
    drwxrwxrwt  17 root root 1269760 Mar  4 10:31 ..
    -rw-rw-r--   1 ale  ale     7804 Mar  4 10:31 .BrokerInfo
    -rw-rw-r--   1 ale  ale        0 Mar  4 10:31 std.err
    -rw-rw-r--   1 ale  ale      624 Mar  4 10:31 std.out
And also:
[ale@cream-15 DataReq]$ cat /tmp/ale_5XbbEhIAtjO5uC5wRyCmyA/std.out
total 32
drwxr-xr-x  2 dteam038 dteam 4096 Mar  4 10:27 .
drwx------  5 dteam038 dteam 4096 Mar  4 10:27 ..
-rw-r--r--  1 dteam038 dteam 7804 Mar  4 10:27 .BrokerInfo
-rw-r--r--  1 dteam038 dteam  216 Mar  4 10:27 calc-pi.sh
-rw-r--r--  1 dteam038 dteam   17 Mar  4 10:27 fileA
-rw-r--r--  1 dteam038 dteam   17 Mar  4 10:27 fileB
-rw-r--r--  1 dteam038 dteam  166 Mar  4 10:27 https_3a_2f_2fdevel17.cnaf.infn.it_3a9000_2f5XbbEhIAtjO5uC5wRyCmyA.output
-rw-r--r--  1 dteam038 dteam    0 Mar  4 10:27 std.err
-rw-r--r--  1 dteam038 dteam    0 Mar  4 10:27 std.out
-rw-------  1 dteam038 dteam    0 Mar  4 10:27 tmp.LesGC30313

  • Check the goodness of the .BrokerInfo file generated:
    • glite-brokerinfo getCE -f glite-brokerinfo getCE -f < .Brokerinfo file path >
      • It should return the matched CE
    • glite-brokerinfo getDataAccessProtocol -f < .Brokerinfo file path >
      • It should return the DataAccessProtocol parameter as specified in the jdl
    • glite-brokerinfo getInputData -f < .Brokerinfo file path >
      • It should return the InputData files specified in the jdl
    • glite-brokerinfo getSEs -f < .Brokerinfo file path >
      • It should return all the SEs (seen by the BDII of the WMS) that store all the files specified in the InputData attribute (You can chek it using the command lcg-lr --vo <VO name> <logical file name>)
    • glite-brokerinfo getCloseSEs -f < .Brokerinfo file path >
      • It should return only the CloseSe to the matched CE (to see them use the command: ldapsearch -x -h <specify the BDII host name> -p 2170 -b mds-vo-name=local,o=grid "(&(objectclass=GlueCESEBindGroup)(GlueCESEBindGroupCEUniqueID=<specify the CE name>))" | grep GlueCESEBindGroupSEUniqueID)
    • glite-brokerinfo getLFN2SFN <LFN of the files in InputData > -f < .Brokerinfo file path >
      • It should return the output of the command lcg-lr --vo <VO name> <logical file name>
    • glite-brokerinfo getVirtualOrganization -f < .Brokerinfo file path >
      • It should return your VO

-- ElisabettaMolinari - 2010-02-24

Edit | Attach | PDF | History: r30 | r9 < r8 < r7 < r6 | Backlinks | Raw View | More topic actions...
Topic revision: r7 - 2010-03-04 - AlessioGianelle
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback