Skip to content

Commit

Permalink
Add inline code examples for the S3 datasource (#29)
Browse files Browse the repository at this point in the history
  • Loading branch information
nikita-marchant authored Aug 4, 2021
1 parent c5f9257 commit 2d98dee
Show file tree
Hide file tree
Showing 3 changed files with 44 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -139,5 +139,21 @@
{% endfor %}
</dd>
</div>
<div class="sm:col-span-2">
<dt class="text-sm font-medium text-gray-500">
{% translate "Usage in Python" %}
</dt>
<dd class="mt-1 text-sm text-gray-900">
<pre class="rounded bg-black text-white p-5">{% include "connector_s3/partials/datasource_demo.py" %}</pre>
</dd>
</div>
<div class="sm:col-span-2">
<dt class="text-sm font-medium text-gray-500">
{% translate "Usage in R" %}
</dt>
<dd class="mt-1 text-sm text-gray-900">
<pre class="rounded bg-black text-white p-5">{% include "connector_s3/partials/datasource_demo.r" %}</pre>
</dd>
</div>
</dl>
{% endblock %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Reading and writing to S3 using Pandas
import pandas as pd

df = pd.read_csv("s3://{{datasource.s3_name}}/path-in-bucket/example.csv")
df.to_csv("s3://{{datasource.s3_name}}/other-path-in-bucket/result.csv")


# Using S3FS to work directly with S3 resources
import s3fs, json

fs = s3fs.S3FileSystem()
with fs.open("s3://{{datasource.s3_name}}/path-in-bucket/example.json", "rb") as f:
print(len(json.load(f)))
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
library(aws.s3)

# Read CSV file
df <- s3read_using(
FUN = read.csv,
object = 's3://{{ datasource.s3_name}}/path-in-bucket/example.csv'
)

# Write CSV file
df.out <- head(df)
s3write_using(
x = df,
FUN = write.csv,
object = 's3://{{ datasource.s3_name}}/path-in-bucket/some-output.csv'
)

0 comments on commit 2d98dee

Please sign in to comment.