[8.0] Interpret outputPath with LFN: prefix as an absolute one - #8603
[8.0] Interpret outputPath with LFN: prefix as an absolute one#8603atsareg wants to merge 9 commits into
Conversation
|
Thank you. All seems fine to me. |
fstagni
left a comment
fstagni
left a comment
There was a problem hiding this comment.
@aldbr has added a lot of tests for JobWrapper in https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/WorkloadManagementSystem/JobWrapper/test/Test_JobWrapper.py.
Can you see if (some of) those can be used here? I am always somewhat scared of changing the JobWrapper.
| # Check whether the list of LFNs has globbable patterns | ||
| globbedLfnList = [] | ||
| for lfn in lfnList: | ||
| lfnPath = os.path.dirname(lfn) | ||
| lfnLocal = os.path.basename(lfn) | ||
| globbedLfnList += [os.path.join(lfnPath, gLfn) for gLfn in List.uniqueElements(getGlobbedFiles(lfnLocal))] | ||
| if globbedLfnList != lfnList and globbedLfnList: | ||
| self.log.info("Found a pattern in the output data LFN list, LFNs to upload are:", ", ".join(globbedLfnList)) | ||
| lfnList = globbedLfnList | ||
|
|
||
| # Check whether the list of outputData has a globbable pattern |
There was a problem hiding this comment.
There is a code duplication between these inserted lines and those that follow, care to refactor for simplicity?
| # If output path is given with the LFN: prefix, take it as an absolute path | ||
| elif outputPath.startswith("LFN:"): | ||
| outputPath = outputPath[4:] | ||
| basePath = "" |
There was a problem hiding this comment.
Shouldn't this block be before the previous 2 lines? For the case when outputPath == "LFN:/some/where/some/thing.xyz"
Yes, there are many tests to JobWrapper added to the integration/9.0 . In 8.0 there are just few. Will see if I can add something here |
| - file names with wild cards; same after the file names expansion; | ||
| - file names in a form "LFN:/vo/full/destination/path/file.name"; in this case the file will be uploaded | ||
| to the specified LFN path without taking into account the OutputPath. Note that file.name here can be also | ||
| specified with wild cards. |
There was a problem hiding this comment.
Give an example of a wild card.
| - if given as ``"LFN:/output/path"``, it will be taken as an absolute path for | ||
| output files in the logical namespace. It is the responsibility of the user to make | ||
| sure that this path is accessible for writing for the user's data. |
There was a problem hiding this comment.
I would like to double check this. What if the user is indeed "not prevented" to upload to such location, but there are no effective policies preventing it? For example, is a simple user prevented from specifying "LFN:/lhcb/user/a/anotheruser" ?
There was a problem hiding this comment.
What does prevent it now, if anything ? We've been getting around this restriction for ever by using dirac-dms-add-file directly.
There was a problem hiding this comment.
I am not trying to fix the whole loose security system of the grid, because as we know "the tokens will solve that" ™️
But at least we can try to fix one such use case server side.
There was a problem hiding this comment.
If you are feeling ambitious ;-). But I have never seen an incident like that.
| 3. If multiple output SEs are specified, they will be tried one-by-one for each | ||
| output file until a successful file upload. |
There was a problem hiding this comment.
Please clarify this: does it mean that it will be uploaded (eventually) to all SEs, or it's "done" after the first upload?
5 participants
This PR allows to define absolute outputPath by specifying it with the LFN: prefix.
It also allows using wild cards in outputData file names specified as LFNs
BEGINRELEASENOTES
*WorkloadManagement
NEW: JobWrapper - interpret outputPath with LFN: prefix as an absolute one
NEW: JobWrapper - allow wild cards in output LFNs
For examples look into release.notes
ENDRELEASENOTES