Invoke-KustoIngestFromStream.md

external help file	Module Name	online version	schema
PowerShellKusto.dll-Help.xml	PowerShellKusto	https://github.com/santisq/PowerShellKusto/blob/main/docs/en-US/Invoke-KustoIngestFromStream.md	2.0.0

Invoke-KustoIngestFromStream

SYNOPSIS

Ingests Streams into Azure Data Explorer.

SYNTAX

Invoke-KustoIngestFromStream
    [-Stream] <Stream>
    [-Table] <String>
    [-Database <String>]
    [-LeaveOpen]
    [-Mapping <IngestionMapping>]
    [-Format <DataSourceFormat>]
    [-IgnoreFirstRecord]
    [-MaxRetries <Int32>]
    [-RetryDelay <TimeSpan>]
    [<CommonParameters>]

DESCRIPTION

Similar to Invoke-KustoIngestFromStorage, the Invoke-KustoIngestFromStream cmdlet allows you to ingest data into a table on your Azure Data Explorer Cluster, the only difference is the source being a Stream instead of a Path or URI.

EXAMPLES

Example 1: Ingest from a Memory Stream

$req = Invoke-WebRequest 'https://myStorageAccount.blob.core.windows.net/....'
Invoke-KustoIngestFromStream $req.RawContentStream -Table myTable -Database myDb

Example 2: Ingest from a File Stream

$fs = (Get-Item .\myJsonFile.json).OpenRead()
Invoke-KustoIngestFromStream $fs -Table myTable -Database myDb -Format multijson

Note

This example specifies a format of multijson instead json because a JSON Array is classified as multijson, while JSON Lines adhere to the json format. See The JSON format for more details.

PARAMETERS

-Database

This non mandatory parameter determines which Database in your Cluster will be targetted by your ingest command.

Note

If not supplied, the Database used will be the one specified when you called Connect-Kusto.

Type: String
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

-Format

This parameter determines the format of the file to be ingested. The default value is csv.

Type: DataSourceFormat
Parameter Sets: (All)
Aliases:
Accepted values: csv, tsv, scsv, sohsv, psv, txt, raw, tsve, w3clogfile, apacheavro, orc, sstream, parquet, avro, multijson, singlejson, json

Required: False
Position: Named
Default value: csv
Accept pipeline input: False
Accept wildcard characters: False

-IgnoreFirstRecord

This switch indicates that ingestion should ignore the first record of a file. This property is useful for files in CSV and similar formats, if the first record in the file are the column names.

Type: SwitchParameter
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: False
Accept pipeline input: False
Accept wildcard characters: False

-LeaveOpen

This switch specifies that the cmdlet should not dispose the Stream after ingestion.

Type: SwitchParameter
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: False
Accept pipeline input: False
Accept wildcard characters: False

-Mapping

This optional parameter indicates how to map data from the source file to the actual columns in the table. You can define the format value with the relevant mapping type.

To create a new mapping object, checkout New-KustoIngestionMapping and New-KustoColumnMapping documentations.

See data mappings and Class KustoIngestionProperties for more information.

Type: IngestionMapping
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

-MaxRetries

Determines the total retry service calls when there is an ingestion failure. The default retry value is 3.

Type: Int32
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

-RetryDelay

Determines the time to wait before retrying. The default value is 1 second.

Type: TimeSpan
Parameter Sets: (All)
Aliases:

Required: False
Position: Named
Default value: [timespan] '00:00:01'
Accept pipeline input: False
Accept wildcard characters: False

-Stream

Specifies the Stream object to be ingested.

Type: Stream
Parameter Sets: (All)
Aliases:

Required: True
Position: 0
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

-Table

Specifies the database table in your Cluster to ingest into.

Type: String
Parameter Sets: (All)
Aliases:

Required: True
Position: 1
Default value: None
Accept pipeline input: False
Accept wildcard characters: False

CommonParameters

This cmdlet supports the common parameters. For more information, see about_CommonParameters.

INPUTS

None

OUTPUTS

Kusto.Ingest.IngestionStatus

NOTES

RELATED LINKS

Ingest from storage

Data mappings

The JSON format

Stream