Difference: WMProxyDocumentation (4 vs. 5)

Revision 52011-11-16 - FabioCapannini

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Line: 83 to 83
 

0.0.1 Credentials Renewal

The User can store a long-lived certificate that can be used by the WMS in order to renew the lifetime of a standard user certificate proxy (usually valid only for 12 hours). Long-running jobs may run into this limit and fail due to expioration of user proxy. WMProxy can automatically request the registration for renewing the proxy certificate sent by the user. This is done through the attribute "MyProxyServer" inside the submitting JDL. When a Proxy Renewal Registration is requested for a certain Job, the WMProxy registers it to the proxy renewal. From that moment on, credentials for such jobs are granted by renewed certificate credentials. Actually all jobs that uses the same credentials links to only one renewed certificate.
Added:
>
>

0.1 Provided Functionality

The operations provided by WMProxy are listed below:

Version:

  • getVersion - returns server internal version
Job Submission and Control:
  • jobListMatch - finds resources which match job requirements
  • jobSubmit - one shot job submission (job registration + job start)
  • jobRegister - registers a job for submission
  • jobStart - starts a previously registered job
  • jobCancel - cancels a job from WMS
  • jobPurge - clean-up of job local disk space
  • getOutputFileList – returns the URIs of job output files
Sandbox and local disk space:
  • getSandboxDestURI - returns the input sandbox destination URI location for a job
  • getSandboxBulkDestURI - returns the input sandbox destination URIs location for a group of jobs
  • getMaxInputSandboxSize - returns the max size available for input sandbox files
  • getFreeQuota – gets the user available local disk space
  • getTotalQuota - gets the user total local disk assigned space
JDL:
  • getJDL - returns the JDL of a submitted job (original or registered to LB after pre-processing)
Proxy:
  • getProxyReq - gets a request for a Proxy to delegate
  • putProxy - put the delegated Proxy into the server machine
  • getDelegatedProxyInfo - gets information of user delegated proxy
  • getJobProxyInfo - gets information of the delegated proxy used to register/submit a job
Job's File Perusal:
  • enableFilePerusal - enable Job's File Perusal functionality setting the file name list
  • getPerusalFiles - get perusal files previously inserted in the list for perusal functionality (allows inspection of job’s files while the job is running)
ACL:
  • addACLItems – adds jobs GACL file entries
  • getACLItems – gets jobs GACL file entries
  • removeACLItem – removes jobs GACL file entry
Templates:
  • getJobTemplate - gets a template for a simple job
  • getDAGTemplate - gets a template for a DAG
  • getCollectionTemplate - gets a template for a collection
  • getIntParametricJobTemplate - gets a template for a parametric job with integer parameters
  • getStringParametricJobTemplate - gets a template for a parametric job with string parameters
Transfer protocols:
  • getTransferProtocols - get the server currently available file transfer protocols

0.2 Job Submission

In this section we describe the sequence of WMProxy operations and the corresponding actions that need to be performed to accomplish job submission to the WMS, i.e. the delivery of the user’s job request in the WM request queue. Job submission requires both, a description of the job to be executed, and a description of the needed resources. These descriptions are provided with a high-level language called JDL. The actual submission is done by calling in sequence the two service operations jobRegister and jobStart When a jobRegister request arrives to the WMProxy, if the client has the rights to proceed, a job identifier is generated and a set of specific attributes needed by the WMS for handling the request appropriately are inserted in the job description. The requesting user is then mapped to a local user by means of LCMAP so that the job local directories and files can be created with appropriate ownership and permissions. When all the aforementioned steps have been successfully completed, the job with the generated job identifier and the enriched JDL description is registered to the LB and from that point on is uniquely identified and can be followed-up throughout the whole system with its identifier. Afterwards, the job is registered for proxy renewal (if requested) and the job handle is returned to the user. In case the JDL description provided by the user represents a compound request (see next section), i.e., a set of jobs, a tree of identifiers is returned. This tree contains the job identifier of the single jobs composing the group, and a job identifier to identify and manage the entire group. The completion of the WMProxy job registration phase is the sign that the job has been taken into account within the Grid environment and is ready to be actually submitted to Grid resources. This further step is triggered by the invocation of the jobStart operation. During the start operation the provided job identifier is used to check if the corresponding job has the pre-requisites for being started, i.e., it has been previously registered by the same WMProxy service and it has not already been started. Only if both conditions are met or previous job start attempts have failed for some reason the job processing is carried on by further manipulating the job description, decompressing, extracting, locating sandbox files in the appropriate areas and lastly writing the request into the WM requests queue. Compound jobs start requires additional steps to be performed before passing the request to WM. These are the registration to LB of all sub-jobs composing the request and the creation of the job reserved area on the file system for all of them. All auxiliary actions needed for the job to run successfully on the remote resource can be performed in the phase between job registration and job start. Relevant examples could be the preparation of the input files needed by the job at run time including both the job input sandbox files and the data stored on Storage Elements (SE) and registered onto Grid Catalogs. In particular the job input sandbox files can be either located on accessible external file server or have to "travel" together with the job from the client machine to the resource where the job is going to run. In the latter case the needed files have to be uploaded by the user to the WMS node so that they can be then managed by the WMS and made available to the job. WMProxy provides the URI of the files destination location through specific operations. Supported file transfer protocols are gsiFTP and HTTPS. The split of the job submission process into the two well distinct phases, job registration and job start, realizes a sort of two-phase commit: with the former operation the job enters the system remaining registered for a configurable period of time; with the latter one, the job can utilize the system. This gives the user full control over single submission phases; in case of failure single operations can be performed again separately. Moreover it allows performing preparatory activities when the job has already entered the system and moving specific time-consuming processing after the operation response has been provided to the client.

submissionsequence.gif

0.3 Service Interface

The WMProxy service exposes operation through a WSDL interface:

http://web.infn.it/gLiteWMS/index.php/techdoc/howtosandguides

The most important purpose of SOA is to achieve loose coupling among interacting components allowing implementation of autonomous services. The loose coupling is achieved through a small set of simple and ubiquitous interfaces, and a set of descriptive messages. In the interfaces only the generic semantics are described: the necessary semantic needed to anticipate possible server behaviour. These interfaces are theoretically available to the global Internet community. The messages represent the way of communication between server and client, and must be descriptive and non-instructive: the server is the only one responsible for the specific design and implementation of the provided service. The messages must be also extensible to allow changes in the service, if needed. To adhere to the SOA model during design and implementation of the WMProxy, the previous guidelines were adopted trying at the same time to provide good service performances to fulfil the user needs for Quality of Service (QoS). Since the WMProxy is a SOAP Web service, the interface is described through the WSDL. A WSDL file represents the available interface needed by the client to access the service. In this file all the operations (services) provided by the WMProxy are described with their argument list, the return value (or structure) and the fault messages. The WSDL file was written following the WS-I Basic Profile. This profile defines a set of Web service specifications that promote interoperability. WS-I Basic Profile addresses the most common problems that could generally affect interoperability, although conformance to it does not completely guarantee the interoperability of a particular service. The profile defines a basic level of conformance based on artifacts. There are three kinds of artifacts, these are:

  • MESSAGE
  • DESCRIPTION
  • REGDATA
WMProxy takes into account the DESCRIPTION artifact, i.e., the descriptions of types, messages and interfaces with protocols, bindings and access points are included in the WSDL file. Compliance of MESSAGE artifacts is assured by the used SOAP container (gSOAP), while the REGDATA principally refers to service registration and discovery with services like Universal Description, Discovery and Integration (UDDI). The latter aspect has not yet been considered exhaustively as it was not a requirement in the EGEE project where a proprietary service discovery mechanism is used. Besides types, messages and interfaces a WSDL file also describes their protocol, data format bindings and the network access points. An instance of a Web service (or a port, represented inside the WSDL file with the wsdl:port tag) is considered conformant if it produces conformant artifacts, and is capable of consuming conformant artifacts, as appropriate. The profile considers all the instances defined for the service (the ports), and verifies the conformance for any of them. Types of Web services (wsdl:binding and wsdl:portType tags) are considered conformant if they result in conformant instances. These are basic rules completed by profile normative statements. In general, to implement a WS-I compliant service, the profile requires a WSDL file, which is composed of specific parts and both describes the service and follows a mandatory structure. The WMProxy WSDL file defines two different ports to provide WMProxy-specific and delegation services (described through an imported WSDL file). This delegation service is the standard service used throughout all gLite, and it is being modified to be aligned with the GT4 delegation service. WSDL type definition of operation arguments is provided to guarantee the correct behavior of all service operations, when accessed from varied language implementations of the requesting client (interoperability). Programming languages have different ways to treat with operation arguments, and different emitters create specific kinds of stub and skeleton (proxy). In the WSDL file, fault messages are also described. These messages are sent back to the client in case of malformed client requests, or in case of service failure. To handle them with a simple, structured approach, an ad-hoc XML Schema type has been designed: the BaseFaultType. The specific fault complex types are developed to extend the BaseFaultType type, and any fault is handled as a BaseFaultType fault at higher level. Work is on-going to make the BaseFaultType definition fully compliant with the WS-BaseFaults specification.
  -- FabioCapannini - 2011-11-04

Line: 90 to 167
 

Added:
>
>

 
META FILEATTACHMENT attachment="wmproxyintegration.gif" attr="" comment="" date="1321354517" name="wmproxyintegration.gif" path="wmproxyintegration.gif" size="41756" user="FabioCapannini" version="1"
META FILEATTACHMENT attachment="delegationsequence.gif" attr="" comment="" date="1321356590" name="delegationsequence.gif" path="delegationsequence.gif" size="4321" user="FabioCapannini" version="1"
Added:
>
>
META FILEATTACHMENT attachment="submissionsequence.gif" attr="" comment="" date="1321442266" name="submissionsequence.gif" path="submissionsequence.gif" size="13355" user="FabioCapannini" version="1"
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback