Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sender options inflight test fails if None #1330

Merged
merged 3 commits into from
Dec 6, 2024
Merged

sender options inflight test fails if None #1330

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
82b1c7a
options inflight test fails if None
petersilva Dec 5, 2024
bae30a8
correct inflight default to always None
petersilva Dec 5, 2024
d2f2d9a
convert now adds inflight setting when sr3 default is different
petersilva Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/source/Explanation/FileCompletion.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,16 @@ Inflight Table
| NONE |Send `sr3_post(7) <sr3_post.7.rst>`_ |post only when file is complete |
| |by AMQP after file is complete. | |
| | |(Best when available) |
| | - fewer round trips (no renames) | - Default on sr_sarra. |
| | - least overhead / highest speed | - Default on sr_subscribe and sender |
| | | when post_broker is set. |
| | - fewer round trips (no renames) | |
| | - least overhead / highest speed | DEFAULT |
| | | |
+-------------+---------------------------------------+--------------------------------------+
| |Files transferred with a *.tmp* suffix.|sending to most other systems |
| .tmp |When complete, renamed without suffix. |(.tmp support built-in) |
| (Suffix) |Actual suffix is settable. |Use to send to Sundew |
| | | |
| | - requires extra round trips for |(usually a good choice) |
| | rename (a little slower) | - default when no post broker set |
| | - requires extra round trips for | |
| | rename (a little slower) | |
+-------------+---------------------------------------+--------------------------------------+
| |Files transferred to a subdir or dir |sending to some other systems |
| tmp/ |When complete, renamed to parent dir. | |
Expand Down
12 changes: 6 additions & 6 deletions docs/source/Reference/sr3_options.7.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -953,18 +953,18 @@ include config
include another configuration within this configuration.


inflight <string> (default: .tmp or NONE if post_broker set)
------------------------------------------------------------
inflight <string> (default: None)
---------------------------------

The **inflight** option sets how to ignore files when they are being transferred
The **inflight** option sets how to ignore files when they are being transferred
or (in mid-flight betweeen two systems). Incorrect setting of this option causes
unreliable transfers, and care must be taken. See `Delivery Completion <../Explanation/FileCompletion.html>`_
for more details.

The value can be a file name suffix, which is appended to create a temporary name during
the transfer. If **inflight** is set to **.**, then it is a prefix, to conform with
the transfer. If **inflight** is set to **.**, then it is a prefix, to conform with
the standard for "hidden" files on unix/linux.
If **inflight** ends in / (example: *tmp/* ), then it is a prefix, and specifies a
If **inflight** ends in / (example: *tmp/* ), then it is a prefix, and specifies a
sub-directory of the destination into which the file should be written while in flight.

Whether a prefix or suffix is specified, when the transfer is
Expand All @@ -979,7 +979,7 @@ This is the same as setting the **fileAgeMin** setting.

Lastly, **inflight** can be set to *NONE*, which case the file is written directly
with the final name, where the recipient will wait to receive a post notifying it
of the file's arrival. This is the fastest, lowest overhead option when it is available.
of the file's arrival. This is the fastest, lowest overhead option when it is available.
It is also the default when a *post_broker* is given, indicating that some
other process is to be notified after delivery.

Expand Down
14 changes: 7 additions & 7 deletions docs/source/fr/Explication/AssurerLaLivraison.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,18 @@ Tableau de Inflight
| |Fichier envoyé avec le bon nom |Envoyer à Sarracenia, et |
| NONE |message`sr3_post(7) <sr3_post.7.rst>`_ |publié quand le fichier est complet |
| |AMQP après que le transfert. | |
| | | (Meilleur quand disponible) |
| | - moins d´aller-retours | défaut pour sr_sarra. |
| | - plus efficace / vite | défaut sur sr_subscribe et sender |
| | | quand post_broker est spécifié. |
| | | Valeur de défaut |
| | - moins d´aller-retours | |
| | - plus efficace / vite | |
| | | |
+-------------+---------------------------------------+--------------------------------------+
| |avec un suffixe *.tmp*. |Envoi à la plupart des autres systèmes|
| .tmp |Lorsqu'il est complet, renommé au fin |(.tmp intégré) |
| (Suffixe) |Le suffixe réel est réglable. |Utiliser pour envoyer à Sundew. |
| | | |
| | -voyages aller-retour supplémentaires |(généralement un bon choix) |
| | pour renommer (un peu plus lent) | - défaut quand il n´y a pas de |
| | | post_broker |
| | -voyages aller-retour supplémentaires | |
| | pour renommer (un peu plus lent) | |
| | | |
+-------------+---------------------------------------+--------------------------------------+
| |Fichier placés dans un sous-répertoire |Envoi à des systèmes qui n´acceptent |
| tmp/ |ou répertoire. Déplacé au fin de |les suffixes |
Expand Down
4 changes: 2 additions & 2 deletions docs/source/fr/Reference/sr3_options.7.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -946,8 +946,8 @@ include config
inclure une autre configuration dans cette configuration.


inflight <string> (défaut: .tmp ou NONE si post_broker est définit)
-------------------------------------------------------------------
inflight <string> (défaut: None)
--------------------------------

L’option **inflight** définit comment ignorer les fichiers lorsqu’ils sont transférés
ou (en plein vol entre deux systèmes). Un réglage incorrect de cette option provoque des
Expand Down
3 changes: 1 addition & 2 deletions sarracenia/flow/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2439,8 +2439,7 @@ def send(self, msg, options) -> int:
# if renaming used, check that the protocol supports it ...
#=================================

if not hasattr(self.proto[self.scheme],
'rename') and options.inflight.startswith('.'):
if not hasattr(self.proto[self.scheme], 'rename') and options.inflight:
logger.warning("%s, rename not supported" % self.scheme)
inflight = None

Expand Down
12 changes: 11 additions & 1 deletion sarracenia/sr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2784,12 +2784,14 @@ def convert1(self,cfg):

synonyms = sarracenia.config.Config.synonyms
accept_all_seen=False
post_broker_seen=False
inflight_seen=False
acceptUnmatched_explicit=False
pos_args_present=False
with open(v3_config_path, 'w') as v3_cfg:
v3_cfg.write( f'# created by: sr3 convert {cfg}\n')
if component in [ 'shovel', 'winnow' ]:
v3_cfg.write('# topicCopy on is only there for bug-for-bug compat with v2. turn it off if you can.\n')
v3_cfg.write('# topicCopy on is only there for bug-for-bug compat with v2. Leave it off if you can.\n')
v3_cfg.write('#topicCopy on\n')

if component in [ 'sarra', 'sender', 'subscribe' ]:
Expand All @@ -2809,9 +2811,17 @@ def convert1(self,cfg):
k = line[0]
if k in synonyms:
k = synonyms[k]
if k in [ 'post_broker' ]:
post_broker_seen=True
if k in [ 'inflight' ]:
inflight_seen=True
if k in [ 'queueName' ]:
queueName=line[1]

if not inflight_seen and post_broker_seen and component in [ 'sarra', 'sender', 'subscribe' ]:
v3_cfg.write('#sr3 inflight defaults to None, v2 defaulted to .tmp when post_broker set.\n')
v3_cfg.write('inflight .tmp\n')

#2nd re-write pass.
subtopicFound=False
with open(v2_config_path, 'r') as v2_cfg:
Expand Down
Loading