RequestRefactoring

Inheritance

Several things are missing in dia, but my idea is there:

• one py object per SQL table
• execution order == index in Request::subReqs list, updated on every change
• add SubRequest or File => list.append
• status checking is very easy as you don't have to guess what SubRequests types are there
• mechanism for status 'calculation' from aggregated classes (i.e. at least one File is in 'Waiting', so SubRequest.Status is forced to be 'Waiting' too, this also could be propagated higher to Request object)
• same for request finalisation: should be done automatically when all SubRequests are 'Done', possible in client code

The API should be lightweight: all classes that are in should be used only for holding properties, not for doing some clever operations or processing.

Using py-properties??? Why not?

Properties (part of language since 2.2), i.e.:

class Request( object ):
  def __RequestID():
    """ RequestID property defined in one go: fset, fget and doc"""
    doc = "this in RequestID property"
    def fset(self, value):
      ## type conversion and type check in one go
      self.__data__["RequestID"] = long(value)
    def fget(self):
      return self.__data__["RequestID"]
    return locals()
  RequestID = property( **__RequestID() )

so you can do:

>>> req = Request() ## empty object
>>> req.RequestID = 12345 ## setter 
>>> req.RequestID ## getter 
12345
>>> req.RequestID = "54321" ## setter 
>>> req.RequestID ## getter
54321
>>> req.RequestID = "foo" ## will throw ValueError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for long() with base 10: 'foo'

Warning! More sophisticated properties definitions are needed for time stamps (should hold datetime.datetime).

Q: should all properties return S_OK/S_ERROR or just values and exceptions?
Keep in mind that fset is not returning any value whatsoever!!! But this is only a bag object, a container without any extra functionality except state machine.

Comment from Fede: the way you define properties is of course working, but a bit obfuscated, at least to me. Why not using the standard examples (with or without decorators) that you find in http://docs.python.org/2/library/functions.html#property ?

State machines

Statuses are somehow special and should be treated separately from all the other properties. In some cases user shouldn't be allow to modify them, i.e. if SubRequest has got at least one 'Waiting' file, its status cannot be set to 'Done', same for Request with at least one 'Waiting' or 'Queued' SubRequest).

Status propagation should be semi-automatic, i.e. on every change to SubReqFile.Status, its parent (SubRequest) should be notified and if possible update its own status (i.e. checking if all children SubReqFiles are 'Done').

• for Request: (no change to current implementation)

• for SubRequest: new state 'Queued', at a time only one SubRequest is marked as 'Waiting', all the others are 'Done' or 'Queued' - this will save system resources, as the only 'selectable' SubRequests (and hence Requests) are those which are really waiting to be executed (execution order == Request's current execution order)

• for SubReqFile: (no change to current implementation)

Request definition

The new API should allow to build (or modify) a complete Request in an easy way, i.e.:

req = Request() # new request 
## new SubRequest
transfer = SubRequest( requestType='transfer', operation='replicateAndRegister')
## add some file to it 
transfer.addSubReqFile( SubReqFile( LFN="/some/lfn", TargetSE="CERN-FOO,FOO-BAR") )
## add SubRequest to Request
req.addSubRequest( transfer )
## new SubRequest
removal = SubRequest( requestType="removal")
## add some file to it (not preferred way?)
removal += SubReqFile( LFN="/foo/bar" )
## insert removal SubRequest before transfer one, so the execution order will be  'removal', 'transfer'
req.insertBefore( transfer, removal )

BTW it should only allow to append/extend (i.e. add new SubRequest to Request or SubReqFile to SubRequest) and definitely forbid to 'substract' (no way to delete particular SubRequest from its parent Request).

Serialisation

All classes should be able to (de-)serialize themselves to (from) XML and SQL.

XML

• using xml.etree.ElementTree (all node attributes are hold as dict, so setting properties is easy)

SQL

• reading: maybe using DictCursor instead of default (tuple one) - you don't have to guess what is read at particular field TBD!
• writing: using INSERT ... ON DUPLICATE KEY UPDATE syntax (???) or check if ID is defined and if so emit UPDATE, INSERT otherwise.

Validation

Attributes

If possible coded in particular property.fset method (type and value checking).

Overall logic

Need to add some validators, i.e. don't accept request without RequestName set, don't accept adding of SubReqFile to some types of SubRequests.

Validators should be implemented and used in RequestClient. A simple validator is already in: HERE