Skip to content

Commit

Permalink
almost there
Browse files Browse the repository at this point in the history
  • Loading branch information
santisq committed Dec 23, 2024
1 parent 641f901 commit 2f1dc69
Show file tree
Hide file tree
Showing 11 changed files with 102 additions and 56 deletions.
2 changes: 1 addition & 1 deletion docs/en-US/Connect-Kusto.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ This non mandatory parameter determines which Database in your Cluster will be t
> [!TIP]
>
> The `Invoke-` cmdlets also offer supplying this value if it wasn't provided during connection.
> The `Invoke-*` and `Set-*` cmdlets also offer supplying this value if it wasn't provided during connection.

```yaml
Type: String
Expand Down
2 changes: 1 addition & 1 deletion docs/en-US/Invoke-KustoIngestFromStorage.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Ingests local or blob storage files into Azure Data Explorer.
Invoke-KustoIngestFromStorage
[-Path] <String>
[-Table] <String>
[-Database <String>]
[[-Database] <String>]
[-Mapping <IngestionMapping>]
[-Format <DataSourceFormat>]
[-IgnoreFirstRecord]
Expand Down
7 changes: 5 additions & 2 deletions docs/en-US/Invoke-KustoQuery.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,11 @@ Accept wildcard characters: False

### -RequestProperties

Request properties control how a query or command executes and returns results.
For creating a `ClientRequestProperties` object, checkout [`New-KustoClientRequestProperties`](New-KustoClientRequestProperties.md).
Request properties control how a query or command executes and returns results. If no `ClientRequestProperties` object is supplied this cmdlet will use default properties.

> [!NOTE]
>
> You can create new request properties using [New-KustoClientRequestProperties](New-KustoClientRequestProperties.md).

```yaml
Type: ClientRequestProperties
Expand Down
2 changes: 1 addition & 1 deletion docs/en-US/New-KustoColumnMapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Accept wildcard characters: False
### -Properties
Property-bag containing properties specific for each mapping as described in each specific mapping type page.
Property-bag containing properties specific for each mapping as described in each specific mapping type page.
```yaml
Type: Hashtable
Expand Down
72 changes: 55 additions & 17 deletions docs/en-US/Set-KustoBatchingPolicy.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ schema: 2.0.0

## SYNOPSIS

{{ Fill in the Synopsis }}
Sets ingestion batching policy on a Kusto Database or Table.

## SYNTAX

```powershell
Set-KustoBatchingPolicy
[-Table] <String>
[[-Table] <String>]
[[-Database] <String>]
[-MaximumBatchingTimeSpan <TimeSpan>]
[-MaximumNumberOfItems <Int32>]
Expand All @@ -27,23 +27,47 @@ Set-KustoBatchingPolicy

## DESCRIPTION

{{ Fill in the Description }}
The `Set-KustoBatchingPolicy` cmdlet is used to alter the batching policy of a Database or specific Table on an Azure Data Explorer Cluster. See [__Ingestion batching policy__](https://learn.microsoft.com/en-us/kusto/management/batching-policy?view=microsoft-fabric) for more details.

## EXAMPLES

### Example 1
### Example 1: Set new ingestion batching policy for a Table

```powershell
PS C:\> {{ Add example code here }}
$params = @{
MaximumBatchingTimeSpan = '00:00:30'
MaximumNumberOfItems = 500
MaximumRawDataSizeMB = 1024
}
Set-KustoBatchingPolicy -Table myTable -Database myDb @params
```

{{ Add example description here }}
This example alters the ingestion batching policy of `myTable` on `myDb` Database.

### Example 2: Set new ingestion batching policy for a Database

```powershell
$params = @{
MaximumBatchingTimeSpan = '00:00:30'
MaximumNumberOfItems = 500
MaximumRawDataSizeMB = 1024
}
Set-KustoBatchingPolicy -Database myDb @params
```

> [!TIP]
>
> When `-Table` isn't specified, the ingestion batching policy is altered at Database level.
## PARAMETERS

### -Database

{{ Fill Database Description }}
Specifies the name of the Database for which to alter the ingestion batching policy.

> [!NOTE]
>
> If not supplied, the Database used will be the one specified when you called [`Connect-Kusto`](Connect-Kusto.md).
```yaml
Type: String
Expand All @@ -59,7 +83,7 @@ Accept wildcard characters: False
### -MaximumBatchingTimeSpan
{{ Fill MaximumBatchingTimeSpan Description }}
The time limit after which a batch is sealed. __The default value is 5 minutes__.
```yaml
Type: TimeSpan
Expand All @@ -68,14 +92,14 @@ Aliases:

Required: False
Position: Named
Default value: None
Default value: [timespan] '00:05:00'
Accept pipeline input: False
Accept wildcard characters: False
```
### -MaximumNumberOfItems
{{ Fill MaximumNumberOfItems Description }}
The number of files defined as the limit after which a batch is sealed. This setting should only be set in scenarios where you can control the data units, such as blobs or files. In message-based scenarios, such as Event Hubs, IoT Hub, and Azure Cosmos DB change feed, consider using the Time and Size settings to control batching. __The default value is 500 items__.
```yaml
Type: Int32
Expand All @@ -84,14 +108,14 @@ Aliases:

Required: False
Position: Named
Default value: None
Default value: 500
Accept pipeline input: False
Accept wildcard characters: False
```
### -MaximumRawDataSizeMB
{{ Fill MaximumRawDataSizeMB Description }}
The size limit after which a batch is sealed. __The default value is 1024 MB__.
```yaml
Type: Int32
Expand All @@ -100,14 +124,14 @@ Aliases:

Required: False
Position: Named
Default value: None
Default value: 1024
Accept pipeline input: False
Accept wildcard characters: False
```
### -OutputType
{{ Fill OutputType Description }}
Determines the output type this cmdlet will produce. __The default value is `PSObject`__.

```yaml
Type: OutputType
Expand All @@ -124,7 +148,11 @@ Accept wildcard characters: False

### -RequestProperties

{{ Fill RequestProperties Description }}
Request properties control how a query or command executes and returns results. If no `ClientRequestProperties` object is supplied this cmdlet will use default properties.

> [!NOTE]
>
> You can create new request properties using [New-KustoClientRequestProperties](New-KustoClientRequestProperties.md).

```yaml
Type: ClientRequestProperties
Expand All @@ -140,14 +168,18 @@ Accept wildcard characters: False

### -Table

{{ Fill Table Description }}
Specifies the Table for which to alter the ingestion batching policy.

> [!NOTE]
>
> This parameter is optional. If not specified, the ingestion batching policy is altered on Database level.

```yaml
Type: String
Parameter Sets: (All)
Aliases:
Required: True
Required: False
Position: 0
Default value: None
Accept pipeline input: False
Expand All @@ -174,3 +206,9 @@ For more information, see [about_CommonParameters](http://go.microsoft.com/fwlin
## NOTES

## RELATED LINKS

[__Ingestion batching policy__](https://learn.microsoft.com/en-us/kusto/management/batching-policy?view=microsoft-fabric)

[__.alter database policy ingestionbatching command__](https://learn.microsoft.com/en-us/kusto/management/alter-database-ingestion-batching-policy?view=microsoft-fabric)

[__Create a table's ingestion batching policy with the table batching policy wizard__](https://docs.azure.cn/en-us/data-explorer/table-batching-policy-wizard)
18 changes: 13 additions & 5 deletions docs/en-US/Set-KustoIngestionMapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,17 @@ PS C:\> {{ Add example code here }}

{{ Fill Database Description }}

> [!NOTE]
>
> If not supplied, the Database used will be the one specified when you called [`Connect-Kusto`](Connect-Kusto.md).
```yaml
Type: String
Parameter Sets: (All)
Aliases:

Required: False
Position: 2
Position: 1
Default value: None
Accept pipeline input: False
Accept wildcard characters: False
Expand Down Expand Up @@ -83,15 +87,15 @@ Parameter Sets: (All)
Aliases:

Required: True
Position: 0
Position: 2
Default value: None
Accept pipeline input: False
Accept wildcard characters: False
```
### -OutputType
{{ Fill OutputType Description }}
Determines the output type this cmdlet will produce. __The default value is `PSObject`__.

```yaml
Type: OutputType
Expand Down Expand Up @@ -124,7 +128,11 @@ Accept wildcard characters: False

### -RequestProperties

{{ Fill RequestProperties Description }}
Request properties control how a query or command executes and returns results. If no `ClientRequestProperties` object is supplied this cmdlet will use default properties.

> [!NOTE]
>
> You can create new request properties using [New-KustoClientRequestProperties](New-KustoClientRequestProperties.md).

```yaml
Type: ClientRequestProperties
Expand All @@ -148,7 +156,7 @@ Parameter Sets: (All)
Aliases:
Required: True
Position: 1
Position: 0
Default value: None
Accept pipeline input: False
Accept wildcard characters: False
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,6 @@ public sealed class InvokeKustoControlCommandCommand : KustoReaderCommandBase
[Parameter(Mandatory = true, Position = 0)]
public string Command { get; set; } = null!;

[Parameter(Position = 1)]
[ValidateNotNullOrEmpty]
public string? Database { get; set; }

protected override void EndProcessing()
{
try
Expand Down
4 changes: 0 additions & 4 deletions src/PowerShellKusto/Commands/InvokeKustoQueryCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,6 @@ public sealed class InvokeKustoQueryCommand : KustoReaderCommandBase
[Parameter(Mandatory = true, Position = 0)]
public string Query { get; set; } = null!;

[Parameter(Position = 1)]
[ValidateNotNullOrEmpty]
public string? Database { get; set; }

protected override void EndProcessing()
{
try
Expand Down
30 changes: 17 additions & 13 deletions src/PowerShellKusto/Commands/SetKustoBatchingPolicyCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,17 @@ namespace PowerShellKusto.Commands;
[OutputType(typeof(PSObject), typeof(string), typeof(DataTable))]
public sealed class SetKustoBatchingPolicyCommand : KustoReaderCommandBase
{
[Parameter(Mandatory = true, Position = 0)]
public string Table { get; set; } = null!;

[Parameter(Position = 1)]
[Parameter(Position = 0)]
[ValidateNotNullOrEmpty]
public string? Database { get; set; }
public string? Table { get; set; }

[Parameter]
[ValidateTimespan]
public TimeSpan MaximumBatchingTimeSpan { get; set; } = TimeSpan.FromSeconds(10);
public TimeSpan MaximumBatchingTimeSpan { get; set; } = TimeSpan.FromMinutes(5);

[Parameter]
[ValidateRange(1, int.MaxValue)]
public int MaximumNumberOfItems { get; set; } = 100;
public int MaximumNumberOfItems { get; set; } = 500;

[Parameter]
[ValidateRange(1, int.MaxValue)]
Expand All @@ -34,15 +31,22 @@ protected override void EndProcessing()
{
try
{
using ICslAdminProvider client = KustoClientFactory.CreateCslAdminProvider(Builder);
string command = CslCommandGenerator.GenerateTableAlterIngestionBatchingPolicyCommand(
databaseName: Database ?? Builder?.InitialCatalog,
tableName: Table,
ingestionBatchingPolicy: new IngestionBatchingPolicy(
Database ??= Builder?.InitialCatalog;
IngestionBatchingPolicy policy = new(
maximumBatchingTimeSpan: MaximumBatchingTimeSpan,
maximumNumberOfItems: MaximumNumberOfItems,
maximumRawDataSizeMB: MaximumRawDataSizeMB));
maximumRawDataSizeMB: MaximumRawDataSizeMB);

string command = Table is null
? CslCommandGenerator.GenerateDatabaseAlterIngestionBatchingPolicyCommand(
databaseName: Database,
ingestionBatchingPolicy: policy)
: CslCommandGenerator.GenerateTableAlterIngestionBatchingPolicyCommand(
databaseName: Database,
tableName: Table,
ingestionBatchingPolicy: policy);

using ICslAdminProvider client = KustoClientFactory.CreateCslAdminProvider(Builder);
using IDataReader reader = RequestProperties is null
? client.ExecuteControlCommand(command)
: client.ExecuteControlCommand(command, RequestProperties);
Expand Down
13 changes: 5 additions & 8 deletions src/PowerShellKusto/Commands/SetKustoIngestionMappingCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,10 @@ namespace PowerShellKusto.Commands;
public sealed class SetKustoIngestionMappingCommand : KustoReaderCommandBase
{
[Parameter(Mandatory = true, Position = 0)]
public string MappingName { get; set; } = null!;

[Parameter(Mandatory = true, Position = 1)]
public string Table { get; set; } = null!;

[Parameter(Position = 2)]
public string? Database { get; set; }
[Parameter(Mandatory = true, Position = 2)]
public string MappingName { get; set; } = null!;

[Parameter(Mandatory = true)]
public IngestionMapping IngestionMapping { get; set; } = null!;
Expand All @@ -30,7 +27,7 @@ protected override void EndProcessing()
{
try
{
string? database = Database ?? Builder?.InitialCatalog;
Database ??= Builder?.InitialCatalog;
using ICslAdminProvider client = KustoClientFactory.CreateCslAdminProvider(Builder);
string command = CslCommandGenerator.GenerateTableMappingCreateCommand(
mappingKind: IngestionMapping.IngestionMappingKind,
Expand All @@ -40,8 +37,8 @@ protected override void EndProcessing()
removeOldestIfRequired: RemoveOldestIfRequired);

using IDataReader reader = RequestProperties is null
? client.ExecuteControlCommand(database, command)
: client.ExecuteControlCommand(database, command, RequestProperties);
? client.ExecuteControlCommand(Database, command)
: client.ExecuteControlCommand(Database, command, RequestProperties);

HandleReader(reader);
}
Expand Down
4 changes: 4 additions & 0 deletions src/PowerShellKusto/KustoReaderCommandBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ private bool NoHeader
get => _noHeaderParam is { NoHeader.IsPresent: not true };
}

[Parameter(Position = 1)]
[ValidateNotNullOrEmpty]
public string? Database { get; set; }

[Parameter]
public OutputType OutputType { get; set; } = OutputType.PSObject;

Expand Down

0 comments on commit 2f1dc69

Please sign in to comment.