Proxy exception: Unable to get Not Before date from Prox"

WORKAROUND: this change will be overwritten in case of a YAIM reconfiguration

add at the end of the file /etc/profile.d/grid-env.sh the line:

gridenv_set "ICE_DISABLE_DEREGISTER" "1"

restart the glite-wms-ice service:

# /etc/init.d/glite-wms-ice restart

[mcecchi@devel15 ~]$ glite-wms-job-perusal --get -f std.out https://devel09.cnaf.infn.it:9000/JJt1CuAgB4bSQ4mUgoQtMg

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server


Error - WMProxy Server Error
The Operation is not allowed: the job is not started
Error code: SOAP-ENV:Server

WORKAROUND

When getting the above error, you need to rename a file on the WMS:

The WMPROXY guide has been updated: https://edms.cern.ch/document/674643/1

https://wiki.italiangrid.it/twiki/bin/view/WMS/WebHome (WMPROXY guide link changed)

5) Changed documentation

5) Changed documentation

The gLite WMS admin guide has been updated. The WMPROXY admin guide has been updated

1) Supported platforms

2) Major changes

2.1) GLUE2 purchasing and match-making

2.1.1) Purchasing

2.1.2) Match-making with GLUE2.0

2.3) Argus authZ for access control

2.4) Dagmanless DAGs

2.5) Condor 7.8.0

2.6) Support for RFC proxies (bug #88128)

3) Known Issues

4) Code changes

1) Supported platforms

2) Major changes

* 2.1 GLUE2 purchasing and match-making

2.1.1) Purchasing

2.1.2) Match-making with GLUE2.0

* 2.3 Argus authZ for access control

* 2.4 Dagmanless DAGs

* 2.5 Condor 7.8.0

* 2.6 Support for RFC proxies (bug #88128)

3) Known Issues

- ARC CE 12.x is not supperted.

While WMS 3.4 works with ARC CE 11.x, the 12.x releases seem to have broken job submission

- The job perusal feature does not work. Output retrieval fails with this error:

 ===> ERROR: Command glite-wms-job-perusal --get -f out.txt --dir /home/capannini/WMS-Test-Suite/WMS-service/WMSService-Test_20120911165612/jobOutput https://cert-13.cnaf.infn.it:9000/-4AKS693kkIhSsXsCK9xbw failed. Failure message: 
Connecting to the service https://cert-13.cnaf.infn.it:7443/glite_wms_wmproxy_server
Error - WMProxy Server Error
The Operation is not allowed: the job is not started

4) Code changes

Supported platforms

Scientific Linux 5.x x86_64

Scientific Linux 6.x x86_64

Extended Release Notes for WMS 3.4.0

Some examples of match-making:

WMS 3.4.0. Extended Release Notes

2) The entry was NOT already purchased by the GLUE1.3 purchaser. A new insertion of the GLUE 2.0 resource will be done. That entry will only be matched by referring to GLUE2.0 attributes in the JDL requirements and rank expressions.

Match-making with GLUE2.0

- GLUE1.3 and 2.0 attributes are referenced in the JDL through the usual 'other.' operator (see next section), specified in the JDL requirements and rank expressions. Selection on GLUE13 or GLUE20 attributes is simply enabled by referring to the appropriate attributes in the JDL. GLUE20 will be accessed through the structure shown above.

Example:

Match on GLUE13 resources. Same as it has always been:

Requirements = other.GlueCEStateWaitingJobs < 100;
Rank = -other.GlueCEStateWaitingJobs;

Match on GLUE20 resources:

Requirements = other.GLUE2.Computing.Share.WaitingJobs < 100;
Rank = -other.GlueCEStateWaitingJobs;

Match on all GLUE13 and GLUE20 resources:

wj = iff(isdefined(other.GlueCEStateWaitingJobs), other.GlueCEStateWaitingJobs, other.GLUE2.Computing.Share.WaitingJobs)
Requirements = wj < 100;
Rank = -wj;

IMPORTANT: the requirements expressed in the JDL are appended to WmsRequirements, a WMS defined classad expression, that includes authZ checks and various constraints on the resource side, automatically appended in && to the user requirements. The present default for WmsRequirements only includes GLUE13 attributes. So, according to what type of purchaser is enabled by configuration, the WmsRequirements expressions must be adapted accordingly. The default is:

* Dagmanless DAGs

* Support for RFC proxies (bug #88128)

- added ice.proxy_renewal_timeout = 120. This parameter was introduced to add a watchdog to possibly hanging renewal requests sent to myproxy. The parameter defines the duration of the grace period.

- added wm.IiGlueLib (= "libglite_wms_ism_ii_purchaser.so.0";)

- added wm.EnableIsmIiGlue20Purchasing and EnableIsmIiGlue13Purchasing, already mentioned in this document

- added attributes to define LDAP filters for GLUE2.0 II purchasers.

wm.IsmIiG2LDAPCEFilterExt allows to specify a custom selection query to populate the Computing Element data structure.

wm.IsmIiG2LDAPSEFilterExt allows to specify a custom selection query to populate the Storege Element data structure.

Defaults are:

Purchasing

    EnableIsmIiGlue13Purchasing  =  <boolean>;
    EnableIsmIiGlue20Purchasing  =  <boolean>;

To produce a full ASCII dump of the Information System cache, rendered in classad, execute this command

will produce a dump similar to the file has been included in this twiki.

The GLUE 1.3 purchaser is always executed before the GLUE 2.0 one. In this way, when the latter starts fetching entries it can find that the same representation was already produced in GLUE 1.3. So two cases:

Computing Element:

Storage Element:

    [
        info =
            [
                GLUE2 =
                    [
                        Storage =
                            [
                                Endpoint =
                                    [
                                        Semantics =
                                           {
                                             // this is a classad list of strings
                                           };
                                        Implementor = "string";
                                        WSDL =
                                           {
                                             // this is a classad list of strings
                                           };
                                        ServingState = "string";
                                        ID = "string";
                                        HealthState = "string";
                                        SupportedProfile =
                                           {
                                             // this is a classad list of strings
                                           };
                                        Capability =
                                           {
                                             // this is a classad list of strings
                                           };
                                        QualityLevel = "string";
                                        TrustedCA =
                                           {
                                             // this is a classad list of strings
                                           };
                                        ImplementationName = "string";
                                        DowntimeAnnounce = "string";
                                        InterfaceName = "string";
                                        InterfaceExtension =
                                           {
                                              ""
                                           };
                                        OtherInfo =
                                            [
                                            ];
                                        Technology = "string";
                                        IssuerCA = "string";
                                        Policy =
                                           {
                                             // this is a classad list of strings
                                           };
                                        URL = "string";
                                        HealthStateInfo = "string";
                                        ImplementationVersion = "string";
                                        InterfaceVersion = "string";
                                        GLUE2StorageEndpointStorageServiceForeignKey = "string"
                                    ];
                                Manager =
                                    [
                                        ID = "string";
                                        StorageServiceForeignKey = "string";
                                        ProductVersion = "string";
                                        ProductName = "string"
                                    ];
                                Share =
                                    [
                                        Tag = "string";
                                        ServingState = "string";
                                        MaximumLifeTime = <integer>;
                                        ID = "string";
                                        ExpirationMode = "string";
                                        Capacity =
                                            [
                                                TotalSize = <integer>;
                                                Type = "string";
                                                ID = "string";
                                                FreeSize = <integer>;
                                                ReservedSize = <integer>;
                                                UsedSize = <integer>;
                                            ];
                                        StorageServiceForeignKey = "string";
                                        Description = "string";
                                        RetentionPolicy =
                                           {
                                             // this is a classad list of strings
                                           };
                                        DefaultLifeTime = <integer>;
                                        Path = "string";
                                        AccessMode =
                                           {
                                             // this is a classad list of strings
                                           };
                                        AccessLatency = "string";
                                        SharingID = "string"
                                    ]
                            ]
                    ];
                PurchasedBy = "ism_ii_g2_purchaser"
            ]
    ]

Match-making

* Argus authZ for access control

Total added = 6k + 1.5k = 7.5k

Purchasing

- The user requirements are appended to WmsRequirements, that included authenticatin checks and various constraints on the resource side. According to what type of purchaser is selected, the user and wms requirements expressions must be adapted accordingly. They can also be crafted in such a way to be able to manage the co-existence of GLUE 1.3 and 2.0 purchasers. The selection is done with these attributes:

Match-making

To match only GLUE2 resources, replace the default attribute WmsRequirements in the WMS configuration with this expression:

((member(CertificateSubject, other.GLUE2.Computing.Endpoint.Policy) || member(strcat("VO:", VirtualOrganisation), other.GLUE2.Computing.Endpoint.Policy) || FQANmember(strcat("VOMS:", VOMS_FQAN), other.GLUE2.Computing.Endpoint.Policy)) && !FQANmember(strcat("DENY:", VOMS_FQAN), other.GLUE2.Computing.Endpoint.Policy) && (other.Glue2.Computing.Share.MaxRunningJobs == 0 || other.Glue2.Computing.ShareTotalJobs < other.Glue2.Computing.Share.MaxRunningJobs) && (EnableWmsFeedback =?= TRUE ? RegExp("cream", other.GLUE2.Computing.Endpoint.ImplementationName, "i") : true))

to match both GLUE 1.3 and GLUE 2.0, just place the expression above in or with the existing one, kind of: WmsRequirements = (GLUE13Reqs) || (GLUE2Reqs);

WMS 3.4 EMI2 Extended Release Notes

Major changes

* GLUE2 purchasing and match-making

HOW TO TRY IT OUT - User requirements are appended to WmsRequirements, that included authenticatin checks and various constraints on the resource side. According to what type of purchaser is selected, the user and wms requirements expressions must be adapted accordingly. They can also be crafted in such a way to be able to manage the co-existence of GLUE 1.3 and 2.0 purchasers. The selection is done with these attributes:

set

EnableIsmIiGlue13Purchasing = false; EnableIsmIiGlue20Purchasing = true;

in the WM conf, and create a file (the name doesn't matter) like this:

echo "[ command = \"ism_dump\"; ]"> jobdir/new/a

Start the WM, and check that no crash occurs after 3/4 purchasing cycles (decrease the PurchasingRate in case). The dump file will report, for example, entries like this:

    [
        expiry_time = 900;
        update_time = 1340106474;
        id = "ce.csl.ee.upatras.gr:8443/cream-pbs-dteam/";
        info =
            [
                GlueCEAccessControlBaseRule = GLUE2.Computing.Endpoint.Policy;
                GlobusResourceContactString = CEid;
                QueueName = GLUE2.Computing.Share.MappingQueue;
                GlueHostApplicationSoftwareRunTimeEnvironment = GLUE2.ApplicationEnvironment.AppName;
                GlueCEInfoHostName = GLUE2.Computing.Share.OtherInfo.InfoProviderHost;
                GlueCEUniqueID = CEid;
                LRMSType = GLUE2.Computing.Manager.ProductName;
                CEid = "ce.csl.ee.upatras.gr:8443/cream-pbs-dteam";
                GLUE2 =
                    [
                        Computing =
                            [
                                Endpoint =
                                    [
                                        Semantics =
                                           {
                                              "http://wiki.italiangrid.org/twiki/bin/view/CREAM/UserGuide"
                                           };
                                        Implementor = "gLite";
                                        WSDL =
                                           {
                                              "https://ce.csl.ee.upatras.gr:8443/ce-cream/services/CREAM2?wsdl"
                                           };
                                        ServingState = "production";
                                        Name = "ce.csl.ee.upatras.gr_org.glite.ce.CREAM";
                                        ID = "ce.csl.ee.upatras.gr_org.glite.ce.CREAM";
                                        HealthState = "ok";
                                        SupportedProfile =
                                           {
                                              "http://www.ws-i.org/Profiles/BasicProfile-1.0.html"
                                           };
                                        Staging = "staginginout";
                                        Capability =
                                           {
                                              "executionmanagement.jobexecution"
                                           };
                                        QualityLevel = "production";
                                        StartTime = "2012-06-16T16:30:03Z";
                                        TrustedCA =
                                           {
                                              "IGTF"
                                           };
                                        ImplementationName = "CREAM";
                                        DowntimeInfo = "See the GOC DB for downtimes: https://goc.egi.eu/";
                                        InterfaceName = "org.glite.ce.CREAM";
                                        OtherInfo =
                                            [
                                                HostDN = "/C=GR/O=HellasGrid/OU=upatras.gr/CN=ce.csl.ee.upatras.gr";
                                                MiddlewareVersion = "2.0.0-1";
                                                MiddlewareName = "EMI"
                                            ];
                                        Technology = "webservice";
                                        IssuerCA = "/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006";
                                        Policy =
                                           {
                                              "VO:ops",
                                              "VO:dteam",
                                              "VO:see"
                                           };
                                        URL = "https://ce.csl.ee.upatras.gr8443/ce-cream/services";
                                        HealthStateInfo = "/etc/init.d/tomcat5 is already running (25631)";
                                        ImplementationVersion = "1.14";
                                        InterfaceVersion = "2.1";
                                        JobDescription =
                                           {
                                              "glite:jdl"
                                           }
                                    ];
                                Manager =
                                    [
                                        Name = "Computing Manager on ce.csl.ee.upatras.gr";
                                        ID = "ce.csl.ee.upatras.gr_ComputingElement_Manager";
                                        ProductVersion = "2.5.7";
                                        ProductName = "torque"
                                    ];
                                Share =
                                    [
                                        DefaultCPUTime = 2880;
                                        MaxWallTime = 4320;
                                        RunningJobs = 0;
                                        WaitingJobs = 444444;
                                        ServingState = "production";
                                        MaxCPUTime = 2880;
                                        FreeSlots = 0;
                                        EstimatedAverageWaitingTime = 2146660842;
                                        ID = "ops_ops_ce.csl.ee.upatras.gr_ComputingElement";
                                        MappingQueue = "dteam";
                                        TotalJobs = 0;
                                        DefaultWallTime = 60;
                                        Description = "Share of dteam for ops";
                                        OtherInfo =
                                            [
                                                InfoProviderName = "glite-ce-glue2-share-static";
                                                InfoProviderHost = "ce.csl.ee.upatras.gr";
                                                CREAMCEId = "ce.csl.ee.upatras.gr:8443/cream-pbs-dteam";
                                                InfoProviderVersion = "1.0"
                                            ];
                                        Policy =
                                           {
                                              ""
                                           };
                                        EstimatedWorstWaitingTime = 2146660842;
                                        MaxRunningJobs = 999999999
                                    ];
                                Service =
                                    [
                                        Name = "Computing Service ce.csl.ee.upatras.gr_ComputingElement";
                                        Type = "org.glite.ce.CREAM";
                                        ID = "ce.csl.ee.upatras.gr_ComputingElement";
                                        Capability =
                                           {
                                              "executionmanagement.jobexecution"
                                           };
                                        QualityLevel = "production";
                                        Complexity = "endpointType=2, share=2, resource=1"
                                    ]
                       Benchmark =
                           {

                                  [
                                      Name = "Benchmark HEP-SPEC06";
                                      Type = "HEP-SPEC06"; 
                                      ID = "ce.csl.ee.upatras.gr_HEP-SPEC06";
                                      Value = "8.35-HEP-SPEC06"
                                  ],

                                  [
                                      Name = "Benchmark specfp2000";
                                      Type = "specfp2000"; 
                                      ID = "ce.csl.ee.upatras.gr_specfp2000";
                                      Value = 1.763000000000000E+03
                                  ],

                                  [
                                      Name = "Benchmark specint2000";
                                      Type = "specint2000"; 
                                      ID = "ce.csl.ee.upatras.gr_specint2000";
                                      Value = 2.088000000000000E+03
                                  ]
                           }; 
                        ApplicationEnvironment =
                            [
                                AppName =
                                   {
                                      "GLITE-3_0_0",
                                      "GLITE-3_1_0",
                                      "GLITE-3_2_0",
                                      "LCG-2",
                                      "LCG-2_1_0",
                                      "LCG-2_1_1",
                                      "LCG-2_2_0",
                                      "LCG-2_3_0",
                                      "LCG-2_3_1",
                                      "LCG-2_4_0",
                                      "LCG-2_5_0",
                                      "LCG-2_6_0",
                                      "LCG-2_7_0",
                                      "R-GMA",
                                      "vo-see-fortran-4.1.2"
                                   }
                            ]
                    ]; 
                PurchasedBy = "ism_ii_g2_purchaser"
            ]
    ]

Note the attribute PurchasedBy = "ism_ii_g2_purchaser"

- GLUE2.0 match-making PASS

After having enabled G2 purchasing in the WM, as in the above test, created a requirements expression like this in your JDL:

requirements = other.GLUE2.Computing.Endpoint.Name == "ce.csl.ee.upatras.gr_org.glite.ce.CREAM";

where the proper values have been taken from the ISM dump. Then, in the WM section, comment out the WmsRequirements expression and replace it with an expression referring to valid GLUE2.0 attributes (WmsRequirements=true can be good, to start) . Restart the WMPROXY. Now submit a list-match request, making sure that no default is taken by some inherited configurations.

[mcecchi@ui ~]$ glite-wms-job-list-match -a -c glite_wmsui.conf  --endpoint https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server ls_g2.jdl 

Connecting to the service https://devel09.cnaf.infn.it:7443/glite_wms_wmproxy_server

==========================================================================

           COMPUTING ELEMENT IDs LIST 
 The following CE(s) matching your job requirements have been found:

   *CEId*
 - ce.csl.ee.upatras.gr:8443/cream-pbs-dteam
 - ce.csl.ee.upatras.gr:8443/cream-pbs-see

==========================================================================

EnableIsmIiGlue13Purchasing = ; EnableIsmIiGlue20Purchasing == ;

The default is c) EnableIsmIiGlue13Purchasing = true & EnableIsmIiGlue20Purchasing = false:

The GLUE 1.3 purchaser is always executed before the GLUE 2.0 one. In this way, when the latter starts it can find:

c1) esiste già in ISM un G13Ad con id = G2Ad.id: allora si fa semplicemente un C13.Ad.Update(G2Ad) (di fatto merge). La risorsa risultante sarà matchabile con requirements G2 o G13 indipendentemente inoltre, in questo modo, anche la parte di storage continuerà a funzionare purchè eventuali attributi nel JDL riguardanti lo storage siano espressi in G13.

c2) in ISM non c'e' un AD con lo stesso G2Ad.id: in questo caso non facciamo alcun update ma un bel insert dell'ad che sarà matchabile solo con requirements G2 compliant nel JDL.

For example, the authZ check would become:

if(isdefined(other.GlueCEAccessControlBaseRule) , member(CertificateSubject,other.GlueCEAccessControlBaseRule), member(CertificateSubject,other.GLUE2.Computing.Endpoint.Policy));

* Argus authZ for access control

for the moment only openssl old DN format is accepted.

* dagmanless DAGs

Nodes are re-evaluated in the WM every MatchRetryPeriod

* Condor 7.8.0

It will fully enable submission to GRAM5 CEs. The feature requires testing submission to LCG-CE, ARC CE, OSG gram2 and gram5

* refactoring of authN/Z in wmp

requires to test authN/Z at large, not only for Argus

* support for RFC proxies (bug #88128)

LCG-CE not expected to support them. OSG, ARC and CREAM CEs will.

* Use of Cmake replacing autotools

Configuration Changes

- removed DagmanLogLevel

- removed DagmanLogRotate

- removed DagmanMaxPre

- removed MaxDAGRunningNodes

- removed wmp tools

- removed par. bulkMM

- removed par. filelist

- removed par. locallogger in wmp

- removed asynch purchasing

- added IiGlueLib (= "libglite_wms_ism_ii_purchaser.so.0";)

- added EnableIsmIiGlue20Purchasing and EnableIsmIiGlue13Purchasing

- added LDAP filters for g2 ii purchasers, IsmIiG2LDAPCEFilterExt & IsmIiG2LDAPSEFilterExt

- added attributes for .so in dlopen (helper, purchasers)

- par. MatchRetryPeriod reduced 600 -> 300. It now also indicates the time interval with which DAGs are evaluated by the WM engine

- SbRetryDifferentProtocols = true by default in WM conf

- WmsRequirements now has also the queue requirements, that were hard-coded before

Code changes

A lot of code has changed!

Roughly speaking:

~6k lines of legacy code added

[mcecchi@devel10 34]$ find . -name emi1_diff|xargs grep -c -e'^[>].*'
./emi.wms.wms-manager/emi1_diff:1114
./emi.wms.wmproxy/emi1_diff:1754
./emi.wms.wms-brokerinfo/emi1_diff:197
./emi.wms.jobsubmission/emi1_diff:768
./emi.wms.wms-helper/emi1_diff:297
./emi.wms.wms-broker/emi1_diff:36
./emi.wms.ice/emi1_diff:323
./emi.wms.wms-matchmaking/emi1_diff:44
./emi.wms.wms-common/emi1_diff:475
./emi.wms.wms-ism/emi1_diff:383

ISM Purchasers: added another ~1600 (1622) SLOC from new modules

~8k lines of legacy code removed

[mcecchi@devel10 34]$ find . -name emi1_diff|xargs grep -c -e'^[<].*'
./emi.wms.wms-manager/emi1_diff:831  <-- DAG engine
./emi.wms.wmproxy/emi1_diff:2022 <-- authN/Z refactoring, Argus based authZ
./emi.wms.wms-brokerinfo/emi1_diff:200
./emi.wms.jobsubmission/emi1_diff:1464 <-- code refactoring, cleanup for not handling DAGs anymore
./emi.wms.wms-helper/emi1_diff:211
./emi.wms.wms-broker/emi1_diff:23
./emi.wms.ice/emi1_diff:374
./emi.wms.wms-matchmaking/emi1_diff:56
./emi.wms.wms-common/emi1_diff:553
./emi.wms.wms-ism/emi1_diff:307

emi.wms.wms-manager: REMOVED several modules (~2300 lines) once needed to integrate Condor DAGMan

[mcecchi@devel02 src]$ wc -l dagman_helper.* dagman_utils.h filelist_utils.h glite-process-counter-args.c glite-wms-dag-* glite-wms-planner.cpp
  974 dagman_helper.cpp
   72 dagman_helper.h
   38 dagman_utils.h
   58 filelist_utils.h
  375 glite-process-counter-args.c
   48 glite-wms-dag-post.sh
   48 glite-wms-dag-post.sh.in
  769 glite-wms-planner.cpp
 2382 total

-- MarcoCecchi - 2012-06-22