crawlee-one / Exports / PushDataOptions
| Name | Type |
|---|---|
T |
extends object |
- cacheActionOnResult
- cachePrimaryKeys
- cacheStoreId
- datasetId
- filter
- includeMetadata
- io
- log
- maxCount
- pickKeys
- privacyMask
- remapKeys
- requestQueueId
- showPrivate
- transform
• Optional cacheActionOnResult: null | "add" | "remove" | "overwrite"
Define whether we want to add, remove, or overwrite cached entries with results from the actor run
• Optional cachePrimaryKeys: string[]
Define fields that uniquely identify entries for caching
• Optional cacheStoreId: string
ID or name of the key-value store used as cache
• Optional datasetId: string
ID or name of the dataset to which the data should be pushed
• Optional filter: (item: any) => unknown
▸ (item): unknown
Option to filter an entry before pushing it to the dataset.
This serves mainly to allow users to filter the entries from actor input UI.
| Name | Type |
|---|---|
item |
any |
unknown
• Optional includeMetadata: boolean
Whether items should be enriched with request and run metadata.
If truthy, the metadata is set under the metadata property.
• Optional io: CrawleeOneIO<any, any, object>
• Optional log: Log
• Optional maxCount: number
If set, only at most this many entries will be scraped.
The count is determined from the Dataset that's used for the crawler run.
This means that if maxCount is set to 50, but the
associated Dataset already has 40 items in it, then only 10 new entries
will be saved.
• Optional pickKeys: string[]
Option to select which keys (fields) of an entry to keep (discarding the rest) before pushing the entries to the dataset.
This serves mainly to allow users to select the keys from actor input UI.
This is done before remapKeys.
Keys can be nested, e.g. "someProp.value[0]". Nested path is
resolved using Lodash.get().
• privacyMask: PrivacyMask<T>
Determine which properties are considered personal data.
See PrivacyMask.
• Optional remapKeys: Record<string, string>
Option to remap the keys before pushing the entries to the dataset.
This serves mainly to allow users to remap the keys from actor input UI.
Keys can be nested, e.g. "someProp.value[0]". Nested path is
resolved using Lodash.get().
• Optional requestQueueId: string
ID of the RequestQueue that stores remaining requests
• Optional showPrivate: boolean
Whether properties that are considered personal data should be shown as is.
If falsy or not set, these properties are redacted to hide the actual information.
Which properties are personal data is determined by privacyMask.
• Optional transform: (item: any) => any
▸ (item): any
Option to freely transform an entry before pushing it to the dataset.
This serves mainly to allow users to transform the entries from actor input UI.
| Name | Type |
|---|---|
item |
any |
any