forked from schaap/p2p-testframework
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREFERENCE
423 lines (333 loc) · 26.8 KB
/
REFERENCE
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
This document contains all parameters that can be used with the delivered framework. It's main subdivision is by module type (i.e. scenario, file, host, client, etc) and those are divided into the extension modules that exist for that module. Note that when looking for akll parameters of an extension module, you should add the parameters of the parent module as well. So, for example, when looking at host:ssh one can see that hostname, user and port are parameters, but also name and remoteDirectory, etc.
If you're not sure yet how to use these parameters, please read the HOWTO and/or the README first.
A note on passing host, client or file objects as parameters: these are passed by the name of the declared object. An incomplete example:
[file:fakedata]
name=myfile
[execution]
file=myfile
Certain host, client or file modules also support an extension to this name, called an argument selector, which is basically an extra argument telling the object you want not just that object, but a part of that object. This is useful, for example, with the file:fakedata module which can generate multiple files in one object. Using an argument selector one can select one specific file. For example:
[execution]
file=myfile@0
will select the first generated file. Or
[execution]
file=myfile@?
would select one random file. The structure of argument selectors is the name of the object followed by '@' followed by the selector argument, in the above cases '0' and '?'. See the specific modules to find out which argument selectors they support, if any.
= scenario =
- name The name of the scenario. Required.
- file Path to a file on the local machine to be included in the scenario description. Required.
May be specified multiple times, in which case the scenario description is the concatenation of the specified files.
- parallel Set to 'no' to make the scenario be handled sequentially. It will definitely be slower, but this can alleviate
problems with too many threads. Note that the relative timing of the clients is about the same as parallel handling.
Optional, defaults to '' which specified parallel handling.
- timelimit Positive integer number of seconds that specifies the maximum amount of time a scenario is allowed to run.
This limit only goes for the actual running, so from the moment the clients are started they are allowed to run for
this time. Optional, defaults to 600.
- timeout Alternative name of timelimit.
= host =
Note that the parameters of the tc modules are also part of the host object. See the README for more on that.
- name The name of the host object. This name is used to refer to the host object in throughout the scenario.
Usually required (particular extension modules sometimes provide a default)
- remoteDirectory The path to a directory on the remote host which can be used to store temporary files in during the scenario.
Optional, a temporary directory will be created by default (in /tmp usually)
- tc The name of the TC module to load, without any prefixes. E.g. tc=netem to load the tc:netem module
on the host. Optional, empty by default which disables TC. By default tc=netem is provided in the tc:netem
module.
- tcInterface The name of the interface on which traffic control is to be applied. This should be an existing
networking interface on the remote host. Optional, defaults to eth0
- tcMaxDownSpeed Maximum download speed to allow. To be specified in bits per second, possibly postfixed by kbit or mbit.
E.g. tcMaxDownSpeed=10mbit for 10 mbit speeds. Optional, defaults to 0 meaning no restrictions
- tcMaxDownBurst Maximum burst in the download speed. Not allowed if tcMaxDownSpeed is not set. To be specified in bits
per second. Optional, defaults to equal to tcMaxDownSpeed
- tcMaxUpSpeed Like tcMaxDownSpeed, but for upload speed.
- tcMaxUpBurst Like tcMaxDownBurst, but for upload speed.
- tcLossChance Chance to drop a packet. A floating point number between 0.0 and 100.0 inclusive, specifying the chance
as a percentage. Optional, defaults to 0.0
- tcDuplicationChance Chance that a packet will be duplicated. A floating point number between 0.0 and 100.0 inclusive,
specifying the chance as a percentage. Optional, defaults to 0.0
- tcCorruptionChance Chance that a packet will be corrupted. A floating point number between 0.0 and 100.0 inclusive,
specifying the chance as a percentage. Optional, defaults to 0.0
- tcDelay The delay to introduce on each packet in ms, given as a positive integer. Optional, defaults to 0
- tcJitter The maximum deviation on the introduced delay, as set by tcDelay, in ms. Optional, defaults to 0
== host:local ==
Uses the local host, mainly for testing. If you wish to use the local host for serious scenarios consider using host:ssh to 127.0.0.1.
- [none]
== host:ssh ==
Uses a host that can be approached via SSH. This is the preferred way of contacting hosts.
- hostname The hostname of the host to be used over SSH. Hostnames and IP addresses are accepted.
- port The port on which the SSH daemon listens on the host; optional, defaults to 22.
- user The user name to be used for logging in over SSH.
== host:das4 ==
Special handler for those with access to the DAS4 system.
- headnode The hostname of the DAS4 host to SSH to initially. Optional, if left out the module
will try and determine which network you're on and use the local entry node. If
you're not on one of the networks of the institutes hosting DAS4 this will give an
error.
The automated lookup test uses dig -x for a reverse lookup on any inet address found
in ifconfig, except 127.0.0.1, and tries to see if it matches any of the following
networks:
network headnode
------- --------
.vu.nl fs0.das4.vu.nl
.liacs.nl fs1.das4.liacs.nl
.uva.nl fs4.das4.science.uva.nl
.tudelft.nl fs3.das4.tudelft.nl
.astron.nl fs5.das4.astron.nl
Note that fs2.das4.science.uva.nl won't be used automatically.
The above table also holds all valid values for this parameter, unless the
headNodeOverride parameters is set.
- nNodes The number of nodes to request, a positive integer. Optional, defaults to 2.
- reserveTime A positive number of seconds to reserve the nodes; note that you should take into
account that the nodes need to be reserved during setup, so some setup steps can
still occur between reservation and the actual running of the test scenarios. It
is recommended to reserve the nodes for a few minutes more than the maximum
execution time of the scenario. Optional, defaults to 900.
- user The username to use for logging in on the DAS4 system. Required.
- headnodeOverride Set to anything but "" to override the validity checks on the headNode parameter.
Use this for custom headNodes or to bypass DNS lookups by providing the IP of the
headnode. Optional.
- reservation If set this integer is the reservation number to be used for the nodes on the DAS4.
The reservation must already be available when the run starts, will be assumed to be
long enough (reserveTime will be ignored) and should include enough nodes for all
host:das4 instances. The reservation will never be cancelled; this is left to the user.
If any host:das4 instance has a reservation set this will be used. If multiple
host:das4 instances have a reservation set it must be the same.
Selection arguments:
- '?' Select a single host from this DAS4 object. This is equal to '?1'.
- '?n' With n being a positive non-zero integer no larger than nNodes: selects n nodes from
this DAS4 object. Should not be used with anything other than executions: the
execution referring to this name will be duplicated n times, each with a different
random node. If n is 1, this is equal to '?', if n is nNodes this is similar to no
argument selector at all but with the nodes in non-deterministic.
- n The positive zero-based index in the list of nodes of this DAS4 object. Will select
that exact node.
- '' As expected, no selector argument will return the master object. *However*: when used
inside an execution's host argument, this is equal to '?n' with n equal to nNodes but
in deterministic order.
= file =
- name The name of the file object. This name is used to refer to the file object in throughout the scenario.
Required.
- rootHash[xx] A Merkle root hash of the file. Consists of 40 hexadecimal digits. Replace xx in the parameter name with
the chunksize in kbytes upon which the root hash is based (that is the size of the data from which each
leaf hash is calculated). Postfix this by L if you wish to have legacy root hashes, i.e. where the root
is always the 63rd level in the tree. Examples of parameter names: rootHash[1]=... rootHash[8L]=...
Optional, may be specified multiple times but not for the same chunksize.
- metaFile A file with metadata, such as a torrent file. This file will be made available to all client executions,
both seeders and leechers. Should be a path to a file on the command machine. Optional.
== file:none ==
A dummy file object that simply provides no data.
- [none]
== file:local ==
Specifies a local file or directory to use as data.
- path The path to the actual file or directory on the local machine.
- generateTorrent Set this to "yes" to have a torrent file automatically generated from the file;
the torrent file will be uplaoded and its location available through getMetaFile(...).
The metaFile parameter must not be set in this case.
- generateRootHash Set to a chunksize in kbytes to generate root hashes for that chunksize. Chunksizes may
be postfixed with L to generate legacy root hashes. Optional, can be specified multiple
times, requires that the rootHash parameter for the requested chunksize is not set.
- renameFile Set this to "yes" to have the file renamed when uploaded to an automatically generated
name. This is forbidden when automated torent generation is requested. Not valid if
path points to a directory.
== file:remote ==
Specifies a remote file or directory to use as data.
- path The path to the actual file or directory on the remote machine.
- renameFile Set this to "yes" to have the file renamed when uploaded to an automatically generated
name. Not valid if path points to a directory.
== file:fakedata ==
Creates fake data on the remote host that is always the same, non-trivial, of configurable size and real.
- ksize A positive integer, divisible by 4, that denotes the size of the generated file in kbytes. Required.
- binary The path of the remote binary to use. This might be needed when g++ does not work on one of the hosts
this file is used on. Optional, defaults to "" which will have the binary compiled on the fly.
- filename The name of the file that will be created. Optional, defaults to "fakedata".
- multiple The number of fake data files to generate. Optional positive integer, defaults to 1. If a multiple
higher than 1 is specified, the filenames will be "{0}_{1}".format( filename, filecounter ) for
filecounter from 0 to (multiple-1)
- generateTorrent If set to anything but "" the file:fakedata will create torrent meta files for each fake data file
and associate the file:fakedata instances with those meta files. Optional, requires metaFile to not
be set.
- generateRootHash Set to a chunksize in kbytes to generate root hashes for that chunksize. Chunksizes may be postfixed with
L to generate legacy root hashes. The generated root hashes will be associated with the correct
file:fakedata instances. Optional, can be specified multiple times, requires that the rootHash parameter
for the requested chunksize is not set.
- torrentCache Path to a local directory. If set, this directory is taken to contain a cache of torrents for fakedata
files. Each torrent is named '{0}_{1}.torrent'.format( self.size, self.slaveNumber ) (no _{1} if
multiple is 1). Any present torrent files will be used from cache, others will be added to the cache.
Optional, must point to an existing directory. Please note that the filename, if specified, must be
equal to 'fakedata' if a torrent cache is specified: this prevents erroneously named cached torrents.
- rootHashCache Path to a local file. If set, this file is taken to be a root hash cache for fakedata files. The cache
is a binary file containing a pickled python dictionary. Any present root hashes will be used from
cache, others will be added. Optional, must point to a writable (possibly not existing) file.
Selection arguments:
- '?' Will select a random file object from this file:fakedata's collection. Especially useful if multiple > 1
to select a random file from the set. E.g. file=myfile@?
- n The positive zero-based index of the file in the list of files. Allowed ranges [0, multiple). Useful for
selecting one specific file when using multiple > 1. E.g. file=myfile@2 (assumes multiple=3 or larger)
= client =
Please note that the options for builder and source modules are given here as well, since they are also part of the client object. See the README for more on that. Also see below for a short description of the source and builder modules that are provided by default.
- name The name of the client object. This name is used to refer to the client object in throughout the scenario.
Optional, defaults to the name of the extension module used
- extraParameters Extra parameters to be appended on the command line to the client. Client specific. Optional, defaults to ''
- parser The name of the parser object to be used to parse logs from this client. Optional, defaults to a new parser
with the same name as the name of the extension module used; may be specified multiple times
- profile Set this to anything but "" to include external profiling code that will inspect CPU and memory usage every
second, which will be captured in the raw cpu.log. Optional, defaults to ''
- logStart Set this to anything but "" to log the starting time of the client, which will be captured in the raw
starttime.log. Note that this uses the local clock of the remote host. Optional, defaults to ''
It's important to realize the effects of using the local clock: it assumes all clocks of the remote hosts
to be equal, which is theoretically impossible. Always take a small error into account and be sure to check
how much off those clocks are when you see strange things.
- source The name of the source module to load, e.g. source=local to use source:local. Optional, defaults to
source:directory. The values source=local and source=git are also provided by default by the source:local
and source:git modules.
- remoteClient Set to anything but '' to signal that the sources are to be loaded, or found, on the remote host instead of the
commanding host. Optional, defaults to ''
- location The location of the sources. The contents of this parameter depends on the source module used. Required.
- builder The name of the builder module to load, e.g. builder=make to use builder:make. Optional, defaults
to builder:none. The values builder=make and builder=scons are also provided by default by the builder:make
and builder:scons modules.
== client:http ==
Uses lighttpd and aria2 to provided HTTP(S) downloads.
Please note that it is most likely required to use a workload for any non-seeding executions in order
to delay their execution until at least 30 seconds after the seeding executions have started: lighttpd
has some setup time via this module (especially with SSL) which would cause the leechers to connect
and fail too early.
- useSSL : Set to anything but "no" to enable HTTPS instead of HTTP; may be specified multiple
times, last declaration counts; optional, defaults to "no"
- port : Use the specified port for server instances (positive integer < 65536; default 3000)
== client:opentracker ==
Allows running the opentracker BitTorrent tracker software, useful in combination with other BitTorrent clients
- port The port on which the tracker will be listening; required, 1023 < positive int < 65536
- changeTracker The name of a file object for which the metaFile parameter has been set and points
to a .torrent file; the torrent file will be changed to point to the dynamically
retrieved address of the first host running this client; the file object will be
altered to have their metaFile point to the changed torrent file before the files
will be uploaded; can be specified multiple times. Note that the .torrent files to
be altered must have a single tracker set already: the change is based on replacing
the existing single tracker. Please note this does not work with self-multiplying
files. Use changeClientTracker in such cases.
- changeClientTracker The name of a client object for which all associated files are to have their metaFile
parameters checked and, if they point to a .torrent file, the torrent file is to be
updated as if the file object was given as a changeTracker to this object. Note that
the metaFile of that very object will be updated, and hence all clients using that
file object will see the updated meta file.
== client:utorrent ==
Uses the uTorrent binary clients with the webui
- useWine If set to "yes" this will instruct client:utorrent to use the windows client under wine. Note that this requires
the user to make sure wine and xvfb-run function correctly on the target hosts!
- stopWhenSeeding If set to "yes" this will kill the client once the "Seeding" state has been reached. In order to make sure this
goes right, please make sure the string "Seeding" is not to be found in the names of torrents or other
(indirect) parameters of the torrent.
== client:swift ==
Uses the libswift command line client
- listenAddress Which address to listen on (i.e. --listen to swift, together with listenPort)
- listenPort Which port to listen on (i.e. --listen to swift, together with listenAddress)
- tracker Specifies a tracker address (i.e. --tracker to swift); the tracker may be specified as @name or @name:port to
load the named host inside the testing framework after all hosts have been prepared, and use that host's address.
- wait Specified the number of seconds to wait (i.e. --wait to swift in seconds)
- chunkSize Chunk size to be used by the swift process in bytes. Optional, defaults to 1024.
== client:libtorrent ==
Uses the libtorrent mini command line client distributed via https://github.com/schaap/p2p-clients
- [none]
== source:directory ==
Assumes the sources or binaries to be present in the directory pointed to by location; if remoteClient is set this is a directory on the remote host, otherwise on the commanding host
== source:local ==
Assumes the sources or binaries to be present in the directory on the commanding host pointed to by location; if remoteClient is set this means the local sources are first uploaded before the builder starts
== source:git ==
The location is a valid git repository that can be cloned
== builder:none ==
The client has already been built. Compilation is skipped.
== builder:make ==
Uses (GNU) make to build the client
== builder:scons ==
Calls the scons building program to build the client
= execution =
- host The name of the host object on which the execution should run. Required
- client The name of the client object which should be executed. Required
- file The name of the file object which is to be transferred. Optional, may be specified multiple times
- parser The name of the parser object which should parse the logs of this execution.
Optional, can be specified multiple times. See the README for information on how parsers are selected.
- seeder Set to anything but '' to mark this execution as a seeding execution. Optional, defaults to ''
- timeout A non-negative floating point number that indicates a number of seconds to wait before actually starting the
client after the scenario starts. Optional, defaults to 0
- keepSeeding Set to anything but '' to make sure this seeding execution has to end by itself before the scenario ends;
normally seeders are killed when all leechers have finished
- multiply Specify a positive integer number of copies to be created of this execution. Optional, defaults to 1.
The host and file parameters of an execution are the most important places for use of argument selectors.
= workload =
- apply Specifies the name of a client object to apply the workload to; this means every execution of that client
(but see applyToSeeders) will be changed to be included in the generated workload. Optional and may be specified
multiple times. If apply is enver specified then all clients that are part of a (non-seeding) execution are
added as soon as all objects have been loaded.
- applyToSeeders By default a workload generator will only change non-seeding executions. Set this to 'yes' to have it change
seeding executions as well. Optional
- offset Starting time of the simulated workload from the start of the scenario in seconds. Optional, floating point
Please note that workloads do not accept apply parameters with selector arguments.
== workload:linear ==
Creates a division of the clients to arrive at a linear rate
Only one may be specified:
- duration Time in seconds over which the peers should arrive. Arrival rate is calculated.
- rate Arrival rate in number of peers per second. Duration is calculated.
- interval Interval between the arrival of 2 peers in seconds. May be 0. Duration is calculated.
== workload:poisson ==
Creates a division of the clients to arrive like a poisson process
Only one may be specified:
- duration Time in seconds over which the peers should arrive. Arrival rate is calculated.
- rate Arrival rate in number of peers per second. Duration is calculated.
= parser =
- name The name of the parser object. This name will be used to refer to the parser object throughout the scenario.
Optional, defaults to the name of the extension module used
== parser:none ==
A dummy implementation parsing nothing
- [none]
== parser:http ==
A copy of parser:none for easier use with client:http
- [none]
== parser:aria2 ==
The parser for logs from aria2 as retrieved by client:http
- [none]
== parser:lighttpd ==
The parser for logs from lighttpd as retrieved by client:http
- [none]
== parser:opentracker ==
A copy of parser:none for easier use of client:opentracker
- [none]
== parser:utorrent ==
The parser for logs from utorrent as retrieved by client:utorrent
- [none]
== parser:swift ==
The parser for logs from swift as retrieved by client:swift
- [none]
== parser:cpulog ==
A parser for CPU logs as generated by having the profile parameter set on a client
- [none]
== parser:libtorrent ==
The parser for logs from libtorrent as retrieved by client:libtorrent
- [none]
= processor =
- [none]
== processor:savehostname ==
Creates a simple text file for each execution with the name of the host object the execution ran on.
- [none]
== processor:saveisseeder ==
Creates a simple text file for each execution with "YES" in it if the execution was a seeder; "NO" is in it otherwise.
- [none]
== processor:savetimeout ==
Creates a simple text file for each execution with the timeout in seconds (float) before the client was launched.
- [none]
== processor:savefiles ==
Creates a simple text file for each execution with the name sof the files associated with that execution.
- [none]
== processor:gnuplot ==
Runs a given gnuplot script for each parsed log in an attempt to create nice graphs.
- script Path to the fnuplot script to be run
- showErrors Set to 'yes' to have processor:gnuplot show errors found when running gnuplot; these are normally
hidden since it's not uncommon to have gnuplot scripts that can run for only a part of the executions
but hence would spam the log with output. Be sure to enable this while testing new gnuplot scripts.
== processor:statistics ==
Calculates some scenario wide statistics for the leechers and seeders (both memory/CPU related and download related).
- [none]
= viewer =
- [none]
== viewer:htmlcollection ==
Creates an HTML page that describes the whole scenario.
- [none]