-
Notifications
You must be signed in to change notification settings - Fork 675
/
Copy pathNEWS
15657 lines (15306 loc) · 888 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.
* Changes in Slurm 25.05.0rc1
=============================
* Changes in Slurm 24.11.1
==========================
-- With client commands MIN_MEMORY will show mem_per_tres if specified.
* Changes in Slurm 24.11.0
==========================
-- slurmctld - Reject arbitrary distribution jobs that do not specifying a task
count.
-- Fix backwards compatibility of the RESPONSE_JOB_INFO RPC (used by squeue,
scontrol show job, etc.) with Slurm clients version 24.05 and below. This
was a regression in 24.11.0rc1.
-- Do not let slurmctld/slurmd to start if there are more nodes defined in
slurm.conf than the maximum supported amount (64k nodes).
-- slurmctld - Set job's exit code to 1 when a job fails with state
JOB_NODE_FAIL. This fixes "sbatch --wait" not being able to exit with error
code when a job fails for this reason in some cases.
-- Fix certain reservation updates requested from 23.02 clients.
-- slurmrestd - Fix populating non-required object fields of objects as '{}' in
JSON/YAML instead of 'null' causing compiled OpenAPI clients to reject
the response to 'GET /slurm/v0.0.40/jobs' due to validation failure of
'.jobs[].job_resources'.
-- Fix issue where older versions of Slurm talking to a 24.11 dbd could
loose step accounting.
-- Fix minor memory leaks.
-- Fix bad memory reference when xstrchr fails to find char.
-- Remove duplicate checks for a data structure.
-- Fix race condition in stepmgr step completion handling.
-- slurm.spec - add ability to specify patches to apply on the command line
-- slurm.spec - add ability to supply extra version information
-- Fix 24.11 HA issues.
-- Fix requeued jobs keeping their priority until the decay thread happens.
-- Fix potential memory corruption in select/cons_tres plugin.
-- Avoid cache coherency issue on non-x86 platforms that could result in a
POSIX signal being ignored or an abort().
-- slurmctld - Remove assertion in development builds that would trigger if
an outdated client attempted to connect.
-- slurmd - Wait for PrologEpilogTimeout on reconfigure for prologs to finish.
This avoids a situation where the slurmd never detects that the prolog
completed.
-- job_container/tmpfs - Setup x11 forwarding within the namespace.
* Changes in Slurm 24.11.0rc2
=============================
-- slurmctld - fix memory leak when sending a DBD_JOB_START message.
-- Fix issue with accounting rollup dealing with association tables.
-- Fix minor memory leaks.
-- Fix potential thread safety issues.
-- Init mutex in burst_buffer plugins.
-- slurmdbd - don't log errors when no changes occur from db requests.
-- slurmcltd,slurmd - Avoid deadlock during reconfigure if too many POSIX
signals are received.
* Changes in Slurm 24.11.0rc1
=============================
-- Improve error type logged from partial or incomplete reading from socket or
pipe to avoid potentially logging an error from a previous syscall.
-- slurmrestd - Improve the handling of queries when unable to connect to
slurmdbd by providing responses when possible.
-- slurmrestd,sackd,scrun - Avoid rare hangs related to I/O.
-- scrun - Add support '--all' argument for kill subcommand.
-- Remove srun --cpu-bind=rank.
-- Add resource_spec/cpus and resource_spec/memory entry points in data_parser
to print the CpuSpecList and MemSpecLimit in sinfo --json.
-- sinfo - Add '.sinfo[].resource_spec.cpus' and
'.sinfo[].resource_spec.memory' fields to print the CpuSpecList and
MemSpecLimit dumped by 'sinfo --{json|yaml}'.
-- Increase efficiency of sending logs to syslog.
-- Switch to new official YAML mime type "application/yaml" in compliance with
RFC9512 as primary mime type for YAML formatting.
-- slurmrestd - Removed deprecated fields from the following endpoints:
'.result' from 'POST /slurm/v0.0.42/job/submit'.
'.job_id', '.step_id', '.job_submit_user_msg' from 'POST /slurm/v0.0.42/job/{job_id}'.
'.job.exclusive', '.jobs[].exclusive' to 'POST /slurm/v0.0.42/job/submit'.
'.jobs[].exclusive' from 'GET /slurm/v0.0.42/job/{job_id}'.
'.jobs[].exclusive' from 'GET /slurm/v0.0.42/jobs'.
'.job.oversubscribe', '.jobs[].oversubscribe' to 'POST /slurm/v0.0.42/job/submit'.
'.jobs[].oversubscribe' from 'GET /slurm/v0.0.42/job/{job_id}'.
'.jobs[].oversubscribe' from 'GET /slurm/v0.0.42/jobs'.
-- scontrol - Removed deprecated fields '.jobs[].exclusive' and
'.jobs[].oversubscribe' from 'scontrol show jobs --{json|yaml}'.
-- squeue - Removed deprecated fields '.jobs[].exclusive' and
'.jobs[].oversubscribe' from 'squeue --{json|yaml}'.
-- Improve the way to run external commands and fork processes to avoid
non-async-signal safe calls between a fork and an exec. We fork ourselves
now and executes the commands in a safe environment. This includes spank
prolog/epilog executions.
-- Improve MaxMemPerCPU enforcement when exclusive jobs request per node memory
and the partition has heterogeneous nodes.
-- Remove a TOCTOU where multiple steps requesting an energy reading at the
same time could cause too frequent accesses to the drivers.
-- Limit SwitchName to HOST_NAME_MAX chars length.
-- For scancel --ctld and the following rest api endpoints:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
'DELETE /slurm/v0.0.42/jobs'
Support array expressions in the responses to the client.
-- salloc - Always output node names to the user when an allocation is granted.
-- slurmrestd - Removed all v0.0.39 endpoints.
-- select/linear - Reject jobs asking for GRES per job|socket|task or
cpus|mem per GRES.
-- Add /nodes POST endpoint to REST API, supports multiple node update whereas
previously only single nodes could be updated through /node/<nodename>
endpoint:
'POST /slurm/v0.0.42/nodes'
-- Do not allow changing or setting PreemptMode=GANG to a partition as this is
a cluster-wide option.
-- Add "%b" as a file name pattern for the array task id modulo 10.
-- Skip packing empty nodes when they are hidden during REQUEST_NODE_INFO RPC.
-- accounting_storage/mysql - Avoid a fatal condition when the db server is
not reachable.
-- Always lay out steps cyclically on nodes in an allocation.
-- squeue - add priority by partition ('.jobs[].priority_by_partition') to
JSON and YAML output.
-- slurmrestd - Add clarification to "failed to open slurmdbd connection" error
if the error was the result of an authentication failure.
-- Make it so slurmctld responds to RPCs that have authentication errors with
the SLURM_PROTOCOL_AUTHENTICATION_ERROR error code.
-- openapi/slurmctld - Display the correct error code instead of
"Unspecified error" if querying the following endpoints fails:
'GET /slurm/v0.0.40/diag/'
'GET /slurm/v0.0.41/diag/'
'GET /slurm/v0.0.42/diag/'
'GET /slurm/v0.0.40/licenses/'
'GET /slurm/v0.0.41/licenses/'
'GET /slurm/v0.0.42/licenses/'
'GET /slurm/v0.0.40/reconfigure'
'GET /slurm/v0.0.41/reconfigure'
'GET /slurm/v0.0.42/reconfigure'
-- Fix how used cpus are tracked in a job allocation to allow the max number of
concurrent steps to run at a time if threads per core is greater than 1.
-- In existing allocations SLURM_GPUS_PER_NODE environment variable will be
ignored by srun if --gpus is specified.
-- When using --get-user-env explicitly or implicitly, check if PID or mnt
namespaces are disabled and fall back to old logic that does not rely on
them when they are not available.
-- Removed non-functional option SLURM_PROLOG_CPU_MASK from TaskProlog which
was used to reset the affinity of a task based on the mask given.
-- slurmrestd - Support passing of '-d latest' to load latest version of
data_parser plugin.
-- sacct,sacctmgr,scontrol,sdiag,sinfo,squeue,sshare - Change response to
'--json=list' or '--yaml=list' to send list of plugins to stdout and
descriptive header to stderr to allow for easier parsing.
-- slurmrestd - Change response to '-d list', '-a list' or '-s list' to send
list of plugins to stdout and descriptive header to stderr to allow for
easier parsing.
-- sacct,sacctmgr,scontrol,sdiag,sinfo,squeue,sshare,slurmrestd - Avoid crash
when loading data_parser plugins fail due to NULL dereference.
-- Add autodetected gpus to the output of slurmd -C
-- Remove burst_buffer/lua call slurm.job_info_to_string().
-- Add SchedulerParameters=bf_allow_magnetic_slot option. It allows jobs in
magnetic reservations to be planned by backfill scheduler.
-- slurmrestd - Refuse to run as root, SlurmUser, and nobody(99).
-- openapi/slurmctld - Revert regression that caused signaling jobs to cancel
entire job arrays instead of job array tasks:
'DELETE /slurm/v0.0.40/{job_id}'
'DELETE /slurm/v0.0.41/{job_id}'
'DELETE /slurm/v0.0.42/{job_id}'
-- openapi/slurmctld - Support more formats for {job_id} including job steps:
'DELETE /slurm/v0.0.40/{job_id}'
'DELETE /slurm/v0.0.41/{job_id}'
'DELETE /slurm/v0.0.42/{job_id}'
-- Alter scheduling of jobs at submission time to consider job submission time
and job id. This makes it so that that interactive jobs aren't allocated
resources before batch jobs when they have the same priority at submit time.
-- Fix multi-cluster submissions with differing Switch plugins.
-- slurmrestd - Change +prefer_refs flag to default in data_parser/v0.0.42
plugin. Add +minimize_refs flag to inline single referenced schemas in the
OpenAPI schema. This sets the default OpenAPI schema generation behavior of
data_parser/v0.0.42 to match v0.0.41+prefer_refs and v0.0.40 (without
flags).
-- Fix LaunchParameters=batch_step_set_cpu_freq.
-- Clearer seff warning message for running jobs.
-- data_parser/v0.0.42 - Rename JOB_INFO field "minimum_switches" to
"required_switches" to reflect the actual behavior.
-- data_parser/v0.0.42 - Rename ACCOUNT_CONDITION field "association" to
"association" to fix typo.
-- cgroup/v2 - fix cgroup cleanup when running inside a container without write
permissions to /sys/fs/cgroup.
-- cgroup/v2 - fix accounting of swap events detection.
-- Fix gathering MaxRSS for jobs that run shorter than two jobacctgather
intervals. Get the metrics from cgroups memory.peak or
memory.max_usage_in_bytes where available.
-- openapi/slurmctld - Set complex number support for the following fields:
.shares[][].fairshare.factor
.shares[][].fairshare.level
for endpoints:
'GET /slurm/v0.0.42/shares'
and for commands:
sshare --json
sshare --yaml
-- data_parser/v0.0.42 - Avoid dumping "Infinity" for NO_VAL tagged "number"
fields.
-- Add TopologyParam=TopoMaxSizeUnroll=# to allow --nodes=<min>-<max> for
topology/block.
-- sacct - Respect --noheader for --batch-script and --env-vars.
-- sacct - Remove extra newline in output from --batch-script and --env-vars.
-- Add "sacctmgr ping" command to query status of slurmdbd.
-- Generate an error message when a NodeSet name conflicts with a NodeName, and
prevent the controller from starting if such a conflict exists.
-- slurmd - properly detect slurmd restarts in the energy gathering logic
which caused bad numbers in accounting.
-- sackd - retry fetching slurm configs indefinitely in configless mode.
-- job_submit/lua - Add "assoc_qos" attribute to job_desc to display all
potential QOS's for a job's association.
-- job_submit/lua - Add slurm.get_qos_priority() function to retrieve the
given QOS's priority.
-- sbcast - Add --nodelist option to specify where files are transmitted to
-- sbcast - Add --no-allocation option to transmit files to nodes outside of a
job allocation
-- Add DataParserParameters slurm.conf parameter to allow setting default
value for CLI --json and --yaml arguments.
-- seff - improve step's max memory consumption report by using TresUsageInTot
and TresUsageInAve instead of overestimating the values.
-- Enable RPC queueing for REQUEST_KILL_JOBS, which is used when scancel is
executed with --ctld flag.
-- slurmdbd - Add -u option. This is used to determine if restarting the DBD
will result in database conversion.
-- Fix srun inside an salloc in a federated cluster when using IPv6.
-- Calculate the forwarding timeouts according to tree depth rather than
node count / tree width for each level. Fixes race conditions with same
timeouts between two consecutive node levels.
-- Add ability to submit jobs with multiple QOS.
-- Fix difference in behavior when swapping partition order in job submission.
-- Improve PLANNED state detection for mixed nodes and updating state before
yielding backfill locks.
-- Always consider partition priority tiers when deciding to try scheduling
jobs on submit.
-- Prevent starting jobs without reservations on submit when there are pending
jobs with reservations that have flags FLEX or ANY_NODES that can be
scheduled on overlapping nodes.
-- Prevent jobs that request both high and low priority tier partitions from
starting on submit in lower priority tier partitions if it could delay
pending jobs in higher priority tier partitions.
-- scontrol - Wait for slurmctld to start reconfigure in foreground mode
before returning.
-- Improve reconfigure handling on Linux to only close open file descriptors
to avoid long delays on systems with large RLIMIT_NOFILE settings.
-- salloc - Removed --get-user-env option.
-- Removed the instant on feature from switch/hpe_slingshot.
-- Hardware collectives in switch/hpe_slingshot now requires enable_stepmgr.
-- Allow backfill to plan jobs on nodes currently being used by exclusive user
or mcs jobs.
-- Avoid miscaching IPv6 address to hostname lookups that could have caused
logs to have the incorrect hostname.
-- scontrol - Add --json/--yaml support to listpids
-- scontrol - Add liststeps
-- scontrol - Add listjobs
-- slurmrestd - Avoid connection to slurmdbd for the following endpoints:
GET /slurm/v0.0.42/jobs
GET /slurm/v0.0.42/job/{job_id}
-- slurmctld - Changed incoming RPC handling to dedicated thread pool.
-- job_container/tmpfs - Add EntireStepInNS option that will place the
slurmstepd process within the constructed namespace directly.
-- scontrol show topo - Show aggregated block sizes when using topology/block.
-- slurmrestd - Add more descriptive HTTP status for authentication failure
and connectivity errors with controller.
-- slurmrestd - Improve reporting errors from slurmctld for job queries:
'GET /slurm/v0.0.41/{job_id}'
'GET /slurm/v0.0.41/jobs/'
-- Avoid rejecting a step request that needs fewer gres than nodes in the
job allocation.
-- slurmrestd - Tag the never populated '.jobs[].pid' field as deprecated for
the following endpoints:
'GET /slurm/v0.0.42/{job_id}'
'GET /slurm/v0.0.42/jobs/'
-- scontrol,squeue - Tag the never populated '.jobs[].pid' field as deprecated
for the following:
'scontrol show jobs --json'
'scontrol show jobs --yaml'
'scontrol show job ${JOB_ID} --json'
'scontrol show job ${JOB_ID} --yaml'
'squeue --json'
'squeue --yaml'
-- data_parser v0.0.42 - fix timestamp parsing regression introduced in in
v0.0.40 (eaf3b6631f), parsing of non iso 8601 style timestamps
-- cgroup/v2 will detect some special container and namespaced setups and
will work with it.
-- Support IPv6 in configless mode.
-- Add SlurmctldParamters=ignore_constraint_validation to ignore
constraint/feature validation at submission.
-- slurmrestd - Set '.pings[].mode' field as deprecated in the following
endpoints:
'GET /slurm/v0.0.42/ping'
-- scontrol - Set '.pings[].mode' field as deprecated in the following
commands:
'scontrol ping --json'
'scontrol ping --yaml'
-- slurmrestd - Set '.pings[].pinged' field as deprecated in the following
endpoints:
'GET /slurm/v0.0.42/ping'
-- scontrol - Set '.pings[].pinged' field as deprecated in the following
commands:
'scontrol ping --json'
'scontrol ping --yaml'
-- slurmrestd - Add '.pings[].primary' field to the following endpoints:
'GET /slurm/v0.0.42/ping'
-- scontrol - Add '.pings[].primary' field to the following commands:
'scontrol ping --json'
'scontrol ping --yaml'
-- slurmrestd - Add '.pings[].responding' field to the following endpoints:
'GET /slurm/v0.0.42/ping'
-- scontrol - Add '.pings[].responding' field to the following commands:
'scontrol ping --json'
'scontrol ping --yaml'
-- Prevent jobs without reservations from delaying jobs in reservations with
flags FLEX or ANY_NODES in the main scheduler.
-- Fix allowing to ask for multiple different types of TRES when one of
them has a value of 0.
-- slurmctld - Add a grace period to ensure the agent retry queue is properly
flushed during shutdown.
-- Removed src/slurmrestd/plugins/openapi/slurmdbd/openapi.json from source
repository. slurmrest should always be used to generate a new OpenAPI
schema (aka openapi.json or openapi.yaml) after compilation.
-- mpi/pmix - Fix potential deadlock and races with het jobs, and fix potential
memory and FDs leaks.
-- Fix jobs with --gpus being rejected in some edge cases for partitions
where not all nodes have the same amount of GPUs and CPUs configured.
-- In an extra constraints expression in a job request, do not allow an empty
string for a key or value.
-- In an extra constraints expression in a job request, fix validation that
requests are separated by boolean operators.
-- Add TaskPluginParam=OOMKillStep to kill the step as a whole when one task
OOMs.
-- Fix scontrol show conf not showing all TaskPluginParam elements.
-- slurmrestd - Add fields '.job.oom_kill_step' '.jobs[].oom_kill_step' to
'POST /slurm/v0.0.42/job/submit' and 'POST /slurm/v0.0.42/job/allocate'.
-- Improve performance for _will_run_test().
-- Add SchedulerParameters=bf_topopt_enable option to enable experimental hook
to control backfill.
-- If a step fails to launch under certain conditions, set the step's state
to NODE_FAIL.
-- sched/backfill - Fix certain situations where a job would not get a
planned time, which could lead to it being delayed by lower priority jobs.
-- slurmrestd - Dump JSON "null" instead of "{}" (empty object) for
non-required fields in objects to avoid client compatibility issues for
v0.0.42 version tagged endpoints.
-- sacct,sacctmgr,scontrol,sdiag,sinfo,squeue,sshare - Dump "null" instead
"{}" (empty object) for non-required fields in objects to avoid client
compatibility issues when run with --json or --yaml.
* Changes in Slurm 24.05.5
==========================
-- Fix issue signaling cron jobs resulting in unintended requeues.
-- Fix slurmctld memory leak in implementation of HealthCheckNodeState=CYCLE.
-- job_container/tmpfs - Fix SLURM_CONF env variable not being properly set.
-- sched/backfill - Fix job's time_limit being overwritten by time_min for job
arrays in some situations.
-- RoutePart - fix segfault from incorrect memory allocation when node doesn't
exist in any partition.
-- slurmctld - Fix crash when a job is evaluated for a reservation after
removal of a dynamic node.
-- gpu/nvml - Attempt loading libnvidia-ml.so.1 as a fallback for failure in
loading libnvidia-ml.so.
-- slurmrestd - Fix populating non-required object fields of objects as '{}' in
JSON/YAML instead of 'null' causing compiled OpenAPI clients to reject
the response to 'GET /slurm/v0.0.40/jobs' due to validation failure of
'.jobs[].job_resources'.
-- Fix sstat/sattach protocol errors for steps on higher version slurmd's
(regressions since 20.11.0rc1 and 16.05.1rc1 respectively).
-- slurmd - Avoid a crash when starting slurmd version 24.05 with
SlurmdSpoolDir files that have been upgraded to a newer major version of
Slurm. Log warnings instead.
-- Fix race condition in stepmgr step completion handling.
-- Fix slurmctld segfault with stepmgr and MpiParams when running a job array.
-- Fix requeued jobs keeping their priority until the decay thread happens.
* Changes in Slurm 24.05.4
==========================
-- Fix generic int sort functions.
-- Fix user look up using possible unrealized uid in the dbd.
-- Fix FreeBSD compile issue with tls/none plugin.
-- slurmrestd - Fix regressions that allowed slurmrestd to be run as SlurmUser
when SlurmUser was not root.
-- mpi/pmix fix race conditions with het jobs at step start/end which could
make srun to hang.
-- Fix not showing some SelectTypeParameters in scontrol show config.
-- Avoid assert when dumping removed certain fields in JSON/YAML.
-- Improve how shards are scheduled with affinity in mind.
-- Fix MaxJobsAccruePU not being respected when MaxJobsAccruePA is set
in the same QOS.
-- Prevent backfill from planning jobs that use overlapping resources for the
same time slot if the job's time limit is less than bf_resolution.
-- Fix memory leak when requesting typed gres and --[cpus|mem]-per-gpu.
-- Prevent backfill from breaking out due to "system state changed" every 30
seconds if reservations use REPLACE or REPLACE_DOWN flags.
-- slurmrestd - Make sure that scheduler_unset parameter defaults to true even
when the following flags are also set: show_duplicates, skip_steps,
disable_truncate_usage_time, run_away_jobs, whole_hetjob,
disable_whole_hetjob, disable_wait_for_result, usage_time_as_submit_time,
show_batch_script, and or show_job_environment. Additionally, always make
sure show_duplicates and disable_truncate_usage_time default to true when
the following flags are also set: scheduler_unset, scheduled_on_submit,
scheduled_by_main, scheduled_by_backfill, and or job_started. This effects
the following endpoints:
'GET /slurmdb/v0.0.40/jobs'
'GET /slurmdb/v0.0.41/jobs'
-- Ignore --json and --yaml options for scontrol show config to prevent mixing
output types.
-- Fix not considering nodes in reservations with Maintenance or Overlap flags
when creating new reservations with nodecnt or when they replace down nodes.
-- Fix suspending/resuming steps running under a 23.02 slurmstepd process.
-- Fix options like sprio --me and squeue --me for users with a uid greater
than 2147483647.
-- fatal() if BlockSizes=0. This value is invalid and would otherwise cause the
slurmctld to crash.
-- sacctmgr - Fix issue where clearing out a preemption list using
preempt='' would cause the given qos to no longer be preempt-able until set
again.
-- Fix stepmgr creating job steps concurrently.
-- data_parser/v0.0.40 - Avoid dumping "Infinity" for NO_VAL tagged "number"
fields.
-- data_parser/v0.0.41 - Avoid dumping "Infinity" for NO_VAL tagged "number"
fields.
-- slurmctld - Fix a potential leak while updating a reservation.
-- slurmctld - Fix state save with reservation flags when a update fails.
-- Fix reservation update issues with parameters Accounts and Users, when
using +/- signs.
-- slurmrestd - Don't dump warning on empty wckeys in:
'GET /slurmdb/v0.0.40/config'
'GET /slurmdb/v0.0.41/config'
-- Fix slurmd possibly leaving zombie processes on start up in configless when
the initial attempt to fetch the config fails.
-- Fix crash when trying to drain a non-existing node (possibly deleted
before).
-- slurmctld - fix segfault when calculating limit decay for jobs with an
invalid association.
-- Fix IPMI energy gathering with multiple sensors.
-- data_parser/v0.0.39 - Remove xassert requiring errors and warnings to have a
source string.
-- slurmrestd - Prevent potential segfault when there is an error parsing an
array field which could lead to a double xfree. This applies to several
endpoints in data_parser v0.0.39, v0.0.40 and v0.0.41.
-- scancel - Fix a regression from 23.11.6 where using both the --ctld and
--sibling options would cancel the federated job on all clusters instead of
only the cluster(s) specified by --sibling.
-- accounting_storage/mysql - Fix bug when removing an association
specified with an empty partition.
-- Fix setting multiple partition state restore on a job correctly.
-- Fix difference in behavior when swapping partition order in job submission.
-- Fix security issue in stepmgr that could permit an attacker to execute
processes under other users' jobs. CVE-2024-48936.
* Changes in Slurm 24.05.3
==========================
-- data_parser/v0.0.40 - Added field descriptions
-- slurmrestd - Avoid creating new slurmdbd connection per request to
'* /slurm/slurmctld/*/*' endpoints.
-- Fix compilation issue with switch/hpe_slingshot plugin.
-- Fix gres per task allocation with threads-per-core.
-- data_parser/v0.0.41 - Added field descriptions
-- slurmrestd - Change back generated OpenAPI schema for
`DELETE /slurm/v0.0.40/jobs/` to RequestBody instead of using parameters
for request. slurmrestd will continue accept endpoint requests via
RequestBody or HTTP query.
-- topology/tree - Fix issues with switch distance optimization.
-- Fix potential segfault of secondary slurmctld when falling back to the
primary when running with a JobComp plugin.
-- Enable --json/--yaml=v0.0.39 options on client commands to dump data using
data_parser/v0.0.39 instead or outputting nothing.
-- switch/hpe_slingshot - Fix issue that could result in a 0 length state file.
-- Fix unnecessary message protocol downgrade for unregistered nodes.
-- Fix unnecessarily packing alias addrs when terminating jobs with a mix of
non-cloud/dynamic nodes and powered down cloud/dynamic nodes.
-- accounting_storage/mysql - Fix issue when deleting a qos that could remove
too many commas from the qos and/or delta_qos fields of the assoc table.
-- slurmctld - Fix memory leak when using RestrictedCoresPerGPU.
-- Fix allowing access to reservations without MaxStartDelay set.
-- Fix regression introduced in 24.05.0rc1 breaking srun --send-libs parsing.
-- Fix slurmd vsize memory leak when using job submission/allocation commands
that implicitly or explicitly use --get-user-env.
-- slurmd - Fix node going into invalid state when using CPUSpecList and
setting CPUs to the # of cores on a multithreaded node
-- Fix reboot asap nodes being considered in backfill after a restart.
-- Fix --clusters/-M queries for clusters outside of a federation when
fed_display is configured.
-- Fix scontrol allowing updating job with bad cpus-per-task value.
-- sattach - Fix regression from 24.05.2 security fix leading to crash.
-- mpi/pmix - Fix assertion when built under --enable-debug.
* Changes in Slurm 24.05.2
==========================
-- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when
more than 10 threads try to get energy at the same time. This prevented
the possibility to get energy from slurmd by any step until slurmd was
restarted, so losing energy accounting metrics in the node.
-- accounting_storage/mysql - Fix issue where new user with wckey did not
have a default wckey sent to the slurmctld.
-- slurmrestd - Prevent slurmrestd segfault when handling the following
endpoints when none of the optional parameters are specified:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
'GET /slurm/v0.0.40/shares'
'GET /slurm/v0.0.41/shares'
'GET /slurmdb/v0.0.40/instance'
'GET /slurmdb/v0.0.41/instance'
'GET /slurmdb/v0.0.40/instances'
'GET /slurmdb/v0.0.41/instances'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.41/job/{job_id}'
-- Fix IPMI energy gathering when no IPMIPowerSensors are specified in
acct_gather.conf. This situation resulted in an accounted energy of 0
for job steps.
-- Fix a minor memory leak in slurmctld when updating a job dependency.
-- scontrol,squeue - Fix regression that caused incorrect values for
multisocket nodes at '.jobs[].job_resources.nodes.allocation' for
'scontrol show jobs --(json|yaml)' and 'squeue --(json|yaml)'.
-- slurmrestd - Fix regression that caused incorrect values for
multisocket nodes at '.jobs[].job_resources.nodes.allocation' to be dumped
with endpoints:
'GET /slurm/v0.0.41/job/{job_id}'
'GET /slurm/v0.0.41/jobs'
-- jobcomp/filetxt - Fix truncation of job record lines > 1024 characters.
-- Fixed regression that prevented compilation on FreeBSD hosts.
-- switch/hpe_slingshot - Drain node on failure to delete CXI services.
-- Fix a performance regression from 23.11.0 in cpu frequency handling when no
CpuFreqDef is defined.
-- Fix one-task-per-sharing not working across multiple nodes.
-- Fix inconsistent number of cpus when creating a reservation using the
TRESPerNode option.
-- data_parser/v0.0.40+ - Fix job state parsing which could break filtering.
-- Prevent cpus-per-task to be modified in jobs where a -c value has been
explicitly specified and the requested memory constraints implicitly
increase the number of CPUs to allocate.
-- slurmrestd - Fix regression where args '-s v0.0.39,dbv0.0.39' and
'-d v0.0.39' would result in 'GET /openapi/v3' not registering as a valid
possible query resulting in 404 errors.
-- slurmrestd - Fix memory leak for dbv0.0.39 jobs query which occurred if the
query parameters specified account, association, cluster, constraints,
format, groups, job_name, partition, qos, reason, reservation, state, users,
or wckey. This affects the following endpoints:
'GET /slurmdb/v0.0.39/jobs'
-- slurmrestd - In the case the slurmdbd does not respond to a persistent
connection init message, prevent the closed fd from being used, and instead
emit an error or warning depending on if the connection was required.
-- Fix 24.05.0 regression that caused the slurmdbd not to send back an error
message if there is an error initializing a persistent connection.
-- Reduce latency of forwarded x11 packets.
-- Add "curr_dependency" (representing the current dependency of the job)
and "orig_dependency" (representing the original requested dependency of
the job) fields to the job record in job_submit.lua (for job update) and
jobcomp.lua.
-- Fix potential segfault of slurmctld configured with
SlurmctldParameters=enable_rpc_queue from happening on reconfigure.
-- Fix potential segfault of slurmctld on its shutdown when rate limiting
is enabled.
-- slurmrestd - Fix missing job environment for SLURM_JOB_NAME,
SLURM_OPEN_MODE, SLURM_JOB_DEPENDENCY, SLURM_PROFILE, SLURM_ACCTG_FREQ,
SLURM_NETWORK and SLURM_CPU_FREQ_REQ to match sbatch.
-- Add missing bash-completions dependency to slurm-smd-client debian package.
-- Fix bash-completions installation in debian packages.
-- Fix GRES environment variable indices being incorrect when only using a
subset of all GPUs on a node and the --gres-flags=allow-task-sharing option
-- Add missing mariadb/mysql client package dependency to debian package.
-- Fail the debian package build early if mysql cannot be found.
-- Prevent scontrol from segfaulting when requesting scontrol show reservation
--json or --yaml if there is an error retrieving reservations from the
slurmctld.
-- switch/hpe_slingshot - Fix security issue around managing VNI access.
CVE-2024-42511.
-- switch/nvidia_imex - Fix security issue managing IMEX channel access.
CVE-2024-42511.
-- switch/nvidia_imex - Allow for compatibility with job_container/tmpfs.
* Changes in Slurm 24.05.1
==========================
-- Fix slurmctld and slurmdbd potentially stopping instead of performing a
logrotate when receiving SIGUSR2 when using auth/slurm.
-- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02.
-- Fix "Could not find group" errors from validate_group() when using
AllowGroups with large /etc/group files.
-- Prevent an assertion in debugging builds when triggering log rotation
in a backup slurmctld.
-- Add AccountingStoreFlags=no_stdio which allows to not record the stdio
paths of the job when set.
-- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field,
which was never usable. Now it explicitly ignores the value and emits a
warning if it is used for the following endpoints:
'POST /slurm/v0.0.39/job/{job_id}'
'POST /slurm/v0.0.39/job/submit'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.40/job/submit'
'POST /slurm/v0.0.41/job/{job_id}'
'POST /slurm/v0.0.41/job/submit'
'POST /slurm/v0.0.41/job/allocate'
-- mpi/pmi2 - Fix communication issue leading to task launch failure with
"invalid kvs seq from node".
-- Fix getting user environment when using sbatch with "--get-user-env" or
"--export=" when there is a user profile script that reads /proc.
-- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but
GresTypes is not configured.
-- Do not log the following errors when AcctGatherEnergyType plugins are used
but a node does not have or cannot find sensors:
"error: _get_joules_task: can't get info from slurmd"
"error: slurm_get_node_energy: Zero Bytes were transmitted or received"
However, the following error will continue to be logged:
"error: Can't get energy data. No power sensors are available. Try later"
-- sbatch, srun - Set SLURM_NETWORK environment variable if --network is set.
-- Fix cloud nodes not being able to forward to nodes that restarted with new
IP addresses.
-- Fix cwd not being set correctly when running a SPANK plugin with a
spank_user_init() hook and the new "contain_spank" option set.
-- slurmctld - Avoid deadlock during shutdown when auth/slurm is active.
-- Fix segfault in slurmctld with topology/block.
-- sacct - Fix printing of job group for job steps.
-- scrun - Log when an invalid environment variable causes the job submission
to be rejected.
-- accounting_storage/mysql - Fix problem where listing or modifying an
association when specifying a qos list could hang or take a very long time.
-- gpu/nvml - Fix gpuutil/gpumem only tracking last GPU in step. Now,
gpuutil/gpumem will record sums of all GPUS in the step.
-- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1.
-- Fix slurmctld crash on a batch job submission with "--nodes 0,...".
-- Fix dynamic IP address fanout forwarding when using auth/slurm.
-- Restrict listening sockets in the mpi/pmix plugin and sattach to the
SrunPortRange.
-- slurmrestd - Limit mime types returned from query to 'GET /openapi/v3' to
only return one mime type per serializer plugin to fix issues with OpenAPI
client generators that are unable to handle multiple mime type aliases.
-- Fix many commands possibly reporting an "Unexpected Message Received" when
in reality the connection timed out.
-- Prevent slurmctld from starting if there is not a json serializer present
and the extra_constraints feature is enabled.
-- Fix heterogeneous job components not being signaled with scancel --ctld and
'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given,
the heterogeneous job components match the given filters, and the
heterogeneous job leader does not match the given filters.
-- Fix regression from 23.02 impeding job licenses from being cleared.
-- Move error to log_flag which made _get_joules_task error to be logged to the
user when too many rpcs were queued in slurmd for gathering energy.
-- For scancel --ctld and the associated rest api endpoints:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
Fix canceling the final array task in a job array when the task is pending
and all array tasks have been split into separate job records. Previously
this task was not canceled.
-- Fix power_save operation after recovering from a failed reconfigure.
-- slurmctld - Skip removing the pidfile when running under systemd. In that
situation it is never created in the first place.
-- Fix issue where altering the flags on a Slurm account (UsersAreCoords)
several limits on the account's association would be set to 0 in
Slurm's internal cache.
-- Fix memory leak in the controller when relaying stepmgr step accounting to
the dbd.
-- Fix segfault when submitting stepmgr jobs within an existing allocation.
-- Added "disable_slurm_hydra_bootstrap" as a possible MpiParams parameter in
slurm.conf. Using this will disable env variable injection to allocations
for the following variables: I_MPI_HYDRA_BOOTSTRAP,
I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS, HYDRA_BOOTSTRAP,
HYDRA_LAUNCHER_EXTRA_ARGS.
-- scrun - Delay shutdown until after start requested. This caused scrun
to never start or shutdown and hung forever when using --tty.
-- Fix backup slurmctld potentially not running the agent when taking over as
the primary controller.
-- Fix primary controller not running the agent when a reconfigure of the
slurmctld fails.
-- slurmd - fix premature timeout waiting for REQUEST_LAUNCH_PROLOG with large
array jobs causing node to drain.
-- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time.
-- jobcomp/elasticsearch - Fix slurmctld memory leak from curl usage
-- acct_gather_profile/influxdb - Fix slurmstepd memory leak from curl usage
-- Fix 24.05.0 regression not deleting job hash dirs after MinJobAge.
-- Fix filtering arguments being ignored when using squeue --json.
-- switch/nvidia_imex - Move setup call after spank_init() to allow namespace
manipulation within the SPANK plugin.
-- switch/nvidia_imex - Skip plugin operation if nvidia-caps-imex-channels
device is not present rather than preventing slurmd from starting.
-- switch/nvidia_imex - Skip plugin operation if job_container/tmpfs
is configured due to incompatibility.
-- switch/nvidia_imex - Remove any pre-existing channels when slurmd starts.
-- rpc_queue - Add support for an optional rpc_queue.yaml configuration file.
-- slurmrestd - Add new +prefer_refs flag to data_parser/v0.0.41 plugin. This
flag will avoid inlining single referenced schemas in the OpenAPI schema.
* Changes in Slurm 24.05.0
==========================
-- Fix regression in rc1 causing power_save thread to spin continuously.
-- Improve ctld_relay shutdown sequence.
-- Fixed 'make distclean' behavior for contribs/perlapi.
-- slurmrestd - Avoid ignoring numerical only endpoints during startup with
older libjson-c due to type parsing mismatching.
-- Reject non-stepmgr job allocations requesting --resv-ports from the ctld.
-- slurmrestd - Add fields '.job.resv_ports' '.jobs[].resv_ports' to
'POST /slurm/v0.0.41/job/submit' and 'POST /slurm/v0.0.41/job/allocate'.
-- slurmstepd - Fix crash when cleaning up on shutdown with
--enable-memory-leak-debug.
-- Fix segfault in switch/hpe_slingshot plugin due to initialization sequence.
-- scrun - Fix regression in rc1 that caused scrun to crash.
-- Prevent unnecessary log statement when free'ing ports.
-- Fix regression in rc1 causing communication problems when sending large
responses from slurmctld.
-- sreport - fix parsing of 'format=Planned' to prevent it from being
misinterpreted as 'PlannedDown'. 'PlannedDown' is now also known as
'PLNDDown' to match what is printed as the column title.
-- topology/block - Always return an error when the segment size does not
match the system or job specification.
-- Add previously missing timers for Prolog and Epilog scripts when RunInJob is
set.
-- Show an error when PrologFlags RunInJob and Serial are used together.
PrologFlags=Serial is not compatible with how RunInJob operates.
-- Fix memory leak on shutdown when using --enable-memory-leak-debug and
freeing cons_tres node usage.
-- Rename src/stepmgr/gres_ctld.[ch] to src/stepmgr/gres_stepmgr.[ch].
-- Fix various cosmetic issues with states in sinfo.
-- slurmrestd - Avoid crash due to associations query.
-- Calculate a job's min_cpus with consideration to --cpus-per-gpu.
-- Fix scancel request when specifying individual array tasks in combination
with filtering options (in both regular and --interactive mode).
-- Enable MaxStepCount in stepmgr.
-- Enable AccountingStorageEnforce=nojobs,nosteps in stepmgr.
-- Add AccountingStorageParameters=max_step_records to limit how many steps
are recorded in the database for each job -- excluding batch, extern, and
interactive steps.
-- switch/hpe_slingshot - allocate VNIs on the controller for stepmgr jobs and
pass to the stepmgr for steps to use.
-- switch/hpe_slingshot - fix assertion when restarting the controller.
-- switch/hpe_slingshot - fix calculation of free vnis when restarting the
controller with running jobs.
-- Improve default job reserve MPI port allocations that use overcommit or do
not specify a task count for stepmgr enabled jobs.
-- Fix a regression in rc1 resulting in scrun occasionally deadlocking when
the --enable-memory-leak-debug configure option was used.
-- topology/default - Prevent segfault in slurmctld on 'scontrol show topo'.
-- slurmrestd - Avoid creating or requiring a connection to slurmdbd for the
'GET /openapi/v3' endpoint, fixing a regression in rc1.
-- scrun - Fix setting and getting environment via SPANK plugins.
-- sview - Fix nodes tab if a node has RestrictedCoresPerGPU configured.
-- slurmrestd - Add --generate-openapi-spec argument.
-- sview - Prevent segfault when retrieving slurmdbd configuration.
-- Avoid canceling rejected heterogeneous jobs without job write lock.
-- Fix slurmctld crash when reconfiguring with a PrologSlurmctld is running.
-- Fix slurmctld crash after a job has been resized.
* Changes in Slurm 24.05.0rc1
=============================
-- Make slurmstepd retry REQUEST_COMPLETE_BATCH_SCRIPT indefinitely.
-- Always load serializer/json when using any data_parser plugins.
-- slurmrestd - Reject single http query with multiple path requests.
-- slurmrestd - Add time/planned field to slurmdb/v0.0.41/job/{job_id}.
-- Improve Power Save's Resume/Suspend rate limiting.
-- slurmrestd - Improve reliability under high memory pressure by closing
connections instead of forcing a fatal exit due to lack of memory.
-- data_parser/v0.0.41 - Avoid aborting when invoking a not implemented
parser.
-- data_parser/v0.0.41 - Fix how nice values are parsed for job submissions.
-- data_parser/v0.0.41 - Fix regression where parsing error did not result in
invalid request being rejected.
-- Print an error message in 'scontrol reboot' when a node reboot request
is ignored due to the current node state.
-- squeue - Add "--notme" option.
-- data_parser/v0.0.41 - change "association.id" to just include the int "id"
rather than include redundant assoc info (cluster, user, partition, account)
that's already included in the "association" object.
-- data_parser/v0.0.41 - Improve parsing of numeric user id.
-- data_parser/v0.0.41 - Improve parsing of numeric group id.
-- slurmrestd - Generated openapi.json will only populate "deprecated" fields
if true. False is the default value and does not require being present.
-- slurmrestd - Populate missing "deprecated" fields in openapi.json.
-- slurmrestd - Corrected deprecated fields in generated openapi.json not
getting populated.
-- slurmrestd - Generated openapi.json will have reduced number of "$ref"
fields. Where there was only 1 reference for the schema, the "$ref" schema
will be directly populated in place.
-- slurmrestd - Rename *_NO_VAL schemas in generated openapi.json to have
_struct and to pass along correct integer sizing when possible.
-- slurmrestd - Correct description fields in generated openapi.json where
descriptions were not present or too generic.
-- Remove support for Cray XC ("cray_aries") systems.
-- Prevent backup slurmctld from taking over if the heartbeat file is
still being updated. Failure to ping may have been due to clock skew.
-- serializer/yaml - Converted to new parsing interface in libyaml to improve
parsing compatibility.
-- Removed TopologyPlugin tree and dragonfly support from select/linear.
If those topology plugins are desired please switch to select/cons_tres.
-- Changed slurmrestd.service to only listen on TCP socket by default.
Environments with existing drop-in units for the service may need
further adjustments to work after upgrading.
-- Fix how gres are allocated per job when using multiple gres types.
-- Log an error when UnkillableStepTimeout is less than five times
MessageTimeout.
-- Avoid step gres dealloc count underflow errors after reconfiguring or
restarting slurmctld.
-- Fix controller not validating periodic dynamic future registrations.
-- Fix dynamic future nodes registering as new node when specifying -N<name>.
-- Fix sbcast (or srun --bcast) --send-libs when it is used multiple times in
the same job. Previously, subsequent calls to sbcast --send-libs would
overwrite the libraries for the first executable.
-- Add support for sbcast --preserve when job_container/tmpfs configured
(previously documented as unsupported).
-- Changed the default value for UnkillableStepTimeout to 60 seconds or five
times the value of MessageTimeout, whichever is greater.
-- slurmctld - Check if --deadline has been reached and not satisfied on held
jobs, otherwise they could remain without automatic cancellation until after
the job is released.
-- scrun/slurmrestd/sackd - Avoid closing all listening sockets when
interrupted from signal such as SIGALRM. Normal shutdown remains unaffected.
-- Remove systemd AbandonScope() logic for scope units as it is not needed.
-- Fix GresUsed output from `scontrol show nodes --details` showing GRES types
that are not configured on a node.
-- slurmrestd - Fatal during start up when loading content plugin fails.
-- slurmrestd - Reduce complexity in URL path matching.
-- data_parser/v0.0.41 - Emit a warning instead of an error if a disabled
parser is invoked.
-- Federation - allow client command operation when slurmdbd is unavailable.
-- Enforce mutual exclusivity of --systemd and -D when launching daemons
-- slurmctld - remove -d option
-- burst_buffer/lua - Trigger a burst_buffer event for strigger when the
real_size function fails.
-- burst_buffer/lua - Added two new hooks: slurm_bb_test_data_in and
slurm_bb_test_data_out. The syntax and use of the new hooks are documented
in etc/burst_buffer.lua.example. These are required to exist. slurmctld now
checks on startup if the burst_buffer.lua script loads and contains all
required hooks; slurmctld will exit with a fatal error if this is not
successful. Added PollInterval to burst_buffer.conf. Removed the arbitrary
limit of 512 copies of the script running simultaneously.
-- sackd/slurmrestd/scrun - Avoid using empty string while logging unix socket
connections from a listening connection.
-- Fix 20 character username limit from 'sacctmgr show events'
-- Log an error if UsePss or NoShare are configured with a plugin other than
jobacct_gather/linux. In such case these parameters are ignored.
-- helpers.conf - Added Flags=rebootless parameter allowing feature changes
without rebooting compute nodes.
-- scontrol - Add new subcommand 'power' for node power control.
-- data_parser/v0.0.41 - Implement parser of distribution for
/slurm/v0.0.41/job/submit.
-- data_parser/v0.0.41 - Change distribution_plane_size field type from UINT16
to UINT16_NO_VAL for /slurm/v0.0.41/job/submit.
-- topology/block - Replaced the BlockLevels with BlockSizes in topology.conf.
-- Fix slurmd cgroup/v2 startup race with systemd and cgroupfs.
-- Add SystemdTimeout= parameter in cgroup.conf.
-- Add QOS limit MaxTRESRunMinsPerAccount.
-- Add QOS limit MaxTRESRunMinsPerUser.
-- jobcomp/{elasticsearch,kafka} - Send priority alongside the rest of fields.
-- Add contain_spank option to SlurmdParameters. When set, spank_user_init(),
spank_task_post_fork(), and spank_task_exit() will execute within the
job_container/tmpfs plugin namespace.
-- Update job reason appropriately when bf_licenses is used.
-- slurmrestd - Tagged `script` field as deprecated in
'POST /slurm/v0.0.41/job/submit' in anticipation of removal in future
OpenAPI plugin versions.
-- Fix salloc/sbatch/srun crashing with certain invalid nodelist requests.
-- Optimize jobacctgather by not iterating every time over pids that have
already finished.
-- Remote SPANK callbacks invoked by srun get called once instead of twice.
-- auth/slurm - Support multiple keys through slurm.jwks.
-- sched/backfill - Fix issue with bf_continue where a job partition request
could be incorrectly reset back to a partition that is no longer specified
after a job partition update processed during a lock yield time window.
-- slurmrestd - Explicitly set process as dumpable (and ptrace-able) at
startup for systems where suid_dumpable is not 2.
-- slurmrestd - Tag all /slurm/v0.0.39/ and /slurmdb/v0.0.39/ endpoints as
deprecated in anticipation of removal in Slurm 24.11.
-- Add ELIGIBLE environment variable to jobcomp/script plugin.
-- slurmrestd,sackd,scrun - Improve outgoing data efficiency using
non-contiguous write support in kernel.
-- sackd - Add support for SACKD_DEBUG, SACKD_STDERR_DEBUG, and
SACKD_SYSLOG_DEBUG environment variables to control logging.
-- mpi/pmi2 - PMI_process_mapping values have been adapted for executions
where arbitrary distribution/SLURM_HOSTFILE is used. Now it can take into
account multiple instances of the same node inside SLURM_HOSTFILE.
-- Avoid wrong limit oriented (i.e. QosMaxGresPer*) job pending reason for jobs
actually pending on Resources when GPUs are requested per job.
-- Fix --ntasks-per-node not being treated as a max count of tasks per node
when used in combination with --ntasks. --ntasks option will now take
precedence as it is documented.
-- Accept X11 cookies that do not have a display number associated with it.
-- Always use the QOS name for SLURM_JOB_QOS environment variables.
Previously the batch environment would use the description field,
which was usually equivalent to the name.
-- slurmrestd - Add "CRON_JOBS" as possible flag value to the following:
'DELETE /slurm/v0.0.40/jobs' flags field.
'DELETE /slurm/v0.0.41/jobs' flags field.
'DELETE /slurm/v0.0.40/job/{job_id}?flags=' flags query parameter.
'DELETE /slurm/v0.0.41/job/{job_id}?flags=' flags query parameter.
-- Fix ScronParameters=explicit_scancel when using the rest api DELETE jobs
query: if the CRON_JOBS flag is not used then cron jobs will not be
cancelled. The NO_CRON_JOBS flag is ignored in v0.0.40 and removed in
v0.0.41.
-- Pass multi-partition job priorities to job for squeue to display.
-- cgroup/v2 - Require dbus-1 version >= 1.11.16.
-- Add RestrictedCoresPerGPU configuration option.
-- Fix how ntasks is inferred from --cpus-per-task when using --nodes,
--threads-per-core, or --hint=nomultithread.
-- For PreemptMode=CANCEL and PreemptMode=REQUEUE assume that job signalled
for GraceTime was preempted.
-- slurmd - Retry fetching configs indefinitely during startup.
-- Fix SPANK options not bing sent to remote context when --export was used.
-- slurmrestd - Attempt to automatically convert enumerated string arrays with
incoming non-string values into strings. Add warning when incoming value for
enumerated string arrays can not be converted to string and silently ignore
instead of rejecting entire request.
-- slurmrestd - Require `user` and `association_condition` fields to be
populated for requests to 'POST /slurmdb/v0.0.41/users_association'.
-- Allow NodeSet names to be used in SuspendExcNodes.
-- SuspendExcNodes=<nodes>:N now counts allocated nodes in N. The first N
powered up nodes in <nodes> are protected from being suspended.
-- Add SlurmctldParameters=max_powered_nodes=N, which prevents powering up
nodes after the max is reached.
-- Store output, error and input paths in the database and make them available
in accounting tools.
-- slurmrestd - Add 'POST /slurm/v0.0.41/job/allocate' endpoint.
-- Fix issues related to the extern step getting killed before other steps.
This includes the job_containter/tmpfs plugin not cleaning up.
-- Add USER_DELETE reservation flag to allow users with access to a reservation
to delete it.
-- Add CgroupPlugin=disabled to disable any interaction with Cgroups.
-- slurmrestd - Add "STEPMGR_ENABLED" as possible flag value to the following:
'GET /slurm/v0.0.41/jobs' flags field.
'GET /slurm/v0.0.41/job/{job_id}' flags query parameter.
-- scontrol,squeue - Added possible flags "STEPMGR_ENABLED" to '.jobs[].flags' for
'scontrol show jobs --{json|yaml}' and 'squeue --{json|yaml}' responses.
-- Add SlurmctldParameters=enable_stepmgr to enable step management through
the slurmstepd instead of the controller.
-- Avoid slurmstepd infinite loop waiting for tasks termination.
-- Fix logging of JSON/YAML values in some messages where nothing would be
printed as the value instead of the actual JSONified version of the parsed
string.
-- slurmrestd,sackd,scrun - Improve logic around handling kernel provided
buffer size of incoming data in files/sockets/pipes to avoid crashes.
-- Add --segment to job allocation to be used in topology/block.
-- Add --exclusive=topo for use with topology/block.
-- Add ExclusiveTopo to a partition definition in slurm.conf.
-- Add new 'BLOCKED' state to a node.
-- Account coordinators may not increase association job limits above
parent ones
-- Account coordinators can now suspend/resume jobs owned by member users.
-- Add DisableCoordDBD slurmdbd configuration parameter to disable the
coordinator status in all slurmdbd interactions.
-- slurmrestd - Added possible flags "WithAssociations" and "WithCoordinators"
to `.accounts[].flags` for "GET /slurmdb/v0.0.41/accounts/" and
"POST /slurmdb/v0.0.41/accounts/" endpoints.
-- sacctmgr - Added possible flags "WithAssociations" and "WithCoordinators"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- slurmrestd - Rename URL query parameter "with_assocs" to "WithAssociations"
for "GET /slurmdb/v0.0.41/accounts?WithAssociations".
-- slurmrestd - Rename URL query parameter "with_coords" to "WithCoordinators"
for "GET /slurmdb/v0.0.41/accounts?WithCoordinators".
-- slurmrestd - Rename URL query parameter "with_deleted" to "deleted"
for "GET /slurmdb/v0.0.41/accounts?deleted".
-- slurmrestd - Added possible flags "RemoveUsersAreCoords" and
"UsersAreCoords" to `.accounts[].flags` for "GET /slurmdb/v0.0.41/accounts/"
and "POST /slurmdb/v0.0.41/accounts/" endpoints.
-- sacctmgr - Added possible flags "RemoveUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- slurmrestd - Add URL query parameter "UsersAreCoords" and
"RemoveUsersAreCoords" for
"GET /slurmdb/v0.0.41/accounts?UsersAreCoords&RemoveUsersAreCoords".
-- sacctmgr - Add new possible new flags "NoUpdate" and "Exact" to
'.associations[].flags' response from 'sacctmgr show assocs --{json|yaml}'.
-- slurmrestd - Added possible flags "NoUpdate" and "Exact" to
`.associations[].flags` for "GET /slurmdb/v0.0.41/associations/"
and "POST /slurmdb/v0.0.41/associations/" endpoints.
-- Fix false success of REQUEST_FORWARD_DATA RPC that made pmix to get out
of sync during initialization.
-- slurmrestd - Allow startup when slurmdbd is not configured and avoid loading
slurmdbd specific plugins.
-- Added PrologFlags=RunInJob to make prolog and epilog run inside the job
extern step to include it in the job's cgroup.
-- Return '*' for the password field for nss_slurm instead of "x".
-- slurmrestd - Add "topo" as possible value to the following:
'GET /slurm/v0.0.41/jobs' in '.jobs[].shared' field
'GET /slurm/v0.0.41/job/{job_id}' in '.jobs[].shared' field
'POST /slurm/v0.0.41/job/submit' in '.job.shared' and '.jobs[].shared'
'POST /slurm/v0.0.41/job/allocate' in '.job.shared' and '.jobs[].shared'
-- sacctmgr - Added possible flags "NoUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- sacct - Add "topo" as possible value to output of 'sacct --{json|yaml}' to
'.jobs[].shared' field.
-- squeue - Add "topo" as possible value to output of 'squeue --{json|yaml}' to
'.jobs[].shared' field.
-- scontrol - Add "topo" as possible value to output of
'scontrol show jobs --{json|yaml}' to '.jobs[].shared' field.
-- slurmrestd - Add "topo" as possible value to the following:
'GET /slurm/v0.0.41/jobs' in '.jobs[].exclusive' field
'GET /slurm/v0.0.41/job/{job_id}' in '.jobs[].exclusive' field
'POST /slurm/v0.0.41/job/submit' in '.job.exclusive' and
'.jobs[].exclusive'
'POST /slurm/v0.0.41/job/allocate' in '.job.exclusive' and
'.jobs[].exclusive'
-- sacctmgr - Added possible flags "RemoveUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- sacct - Add "topo" as possible value to output of 'sacct --{json|yaml}' to
'.jobs[].exclusive' field.
-- squeue - Add "topo" as possible value to output of 'squeue --{json|yaml}' to
'.jobs[].exclusive' field.
-- scontrol - Add "topo" as possible value to output of
'scontrol show jobs --{json|yaml}' to '.jobs[].exclusive' field.
-- slurmrestd - Add fields '.job.segment_size' and '.jobs[].segment_size' to
'POST /slurm/v0.0.41/job/submit' and 'POST /slurm/v0.0.41/job/allocate'.
-- sacctmgr - Added possible flags "NoUsersAreCoords" and "UsersAreCoords"
to `.associations[].flags` for `sacctmgr show assocs --{json|yaml}`
response.
-- slurmrestd - Added possible flags "NoUsersAreCoords" and "UsersAreCoords" to
`.associations[].flags` for "GET /slurmdb/v0.0.41/associations/"
and "POST /slurmdb/v0.0.41/associations/" endpoints.