Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deno failing with AWS SDK credentials with npm specifier #17666

Closed
soundstep opened this issue Feb 6, 2023 · 11 comments
Closed

Deno failing with AWS SDK credentials with npm specifier #17666

soundstep opened this issue Feb 6, 2023 · 11 comments

Comments

@soundstep
Copy link

This is about using the AWS SDK V3 with Deno and the npm specifier:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/[email protected]';
import { fromContainerMetadata } from 'npm:@aws-sdk/[email protected]';

The issue has reported there: aws/aws-sdk-js-v3#4405.
The import is working properly, but the usage does not.
Can provide help to debug this issue?
Thank you in advance.

@bartlomieju
Copy link
Member

Duplicate of #16810, let's continue there.

@soundstep
Copy link
Author

@bartlomieju I don't think it is the same issue, this is related to credentials. I believe the behavior I reported is related to the following package that is not working with Deno as it is using node-bindings:
https://github.com/awslabs/aws-crt-nodejs
Can we please reopen?

@bartlomieju bartlomieju reopened this Feb 7, 2023
@soundstep
Copy link
Author

Note that the issue with awslabs/aws-crt-nodejs is also why this aws-sdk deno port has been abandoned:
christophgysin/aws-sdk-js-v3#38 (comment)

@soundstep soundstep changed the title Deno failing to use AWS SDK with npm specifier Deno failing with AWS SDK credentials with npm specifier Feb 7, 2023
@bartlomieju
Copy link
Member

@soundstep any chance you could try with latest Deno (v1.34.1) and let me know if the problem persists? We polyfilled a lot of missing APIs since you opened this issue and I expect it should just work these days.

@soundstep
Copy link
Author

Yes, I'll try to check that when I get a chance!

@soundstep
Copy link
Author

soundstep commented Jun 2, 2023

I can confirm that this works with deno 1.34.1 and EC2 container metadata credentials:

#!/usr/bin/env -S deno run -A --reload

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/[email protected]';

console.log('No AWS credentials setup');

const client = new S3Client({
    region: Deno.env.get('AWS_REGION') || 'eu-west-1',
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: '<BUCKET_NAME>',
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

And this was already working with web identity credentials:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/[email protected]';

const client = new S3Client({
    region: Deno.env.get('AWS_REGION') || 'eu-west-1',
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: '<BUCKET_NAME>',
        }),
    );

    console.log('response length:', res.Contents.length);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

@andykais
Copy link

andykais commented Dec 2, 2023

cross-posting another related issue that I came across with the aws sdk and R2 based on their docs for node usage https://developers.cloudflare.com/r2/reference/data-location:

import * as s3 from 'npm:@aws-sdk/client-s3'

const s3_client = new s3.S3({
  region: 'auto',
  endpoint: R2_BASE_URL,
  credentials: {
    accessKeyId: ACCESS_KEY_ID,
    secretAccessKey: ACCESS_KEY_SECRET,
  }
})

const result = await s3_client.createMultipartUpload({
  Bucket: 'MYBUCKET',
  Key: `${Date.now()}`,
})

gives this error:

error: Uncaught (in promise) 400: UnknownError
    at throwDefaultError (file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_CreateMultipartUploadCommandError (file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3276:12)
    at Object.runMicrotasks (ext:core/01_core.js:790:30)
    at processTicksAndRejections (ext:deno_node/_next_tick.ts:53:10)
    at runNextTicks (ext:deno_node/_next_tick.ts:71:3)
    at eventLoopTick (ext:core/01_core.js:184:21)
    at async file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20
    at async file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46

running the same snippet in node (with a few changes like removing the npm: specifier and wrapping the top-level await) will succeed so I am confident this is a deno specific issue. I have also been able to run S3::listObjects in deno, so it seems to be some specific logic that is hit by this method.

@tazmaniax
Copy link

tazmaniax commented Jan 15, 2024

I think this might be related. The following script importing the S3 package from esh.sh and running on Deno 1.39.4 works...

import { S3 } from "https://esm.sh/@aws-sdk/[email protected]"

Deno.serve(async (req) => {
  const { name } = await req.json()

  const config = {
    region: Deno.env.get('AWS_REGION') ?? '',
    credentials: {
      accessKeyId: Deno.env.get('AWS_ACCESS_KEY_ID') ?? '',
      secretAccessKey: Deno.env.get('AWS_SECRET_ACCESS_KEY') ?? '',
      sessionToken: Deno.env.get('AWS_SESSION_TOKEN') ?? '',
    }
  }
  console.log('start instantiating S3 client', config)
  const s3 = new S3(config);
  console.log('finish instantiating S3 client')

  const data = {
    message: `Hello ${name}!`,
  }

  return new Response(
    JSON.stringify(data),
    { headers: { "Content-Type": "application/json" } },
  )
})

However, if S3 is imported using the npm specifier...

// import { S3 } from "https://esm.sh/@aws-sdk/[email protected]"
import { S3 } from "npm:@aws-sdk/[email protected]"

...then the following error occurs

[Info] finish instantiating S3 client

event loop error: TypeError [ERR_INVALID_ARG_TYPE]" argument must be of type string. Received null
    at assertPath (ext:deno_node/path/_util.ts:10:11)
    at join (ext:deno_node/path/_posix.ts:77:5)
    at getCredentialsFilepath (file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/shared-ini-file-loader/2.2.8/dist-cjs/getCredentialsFilepath.js:7:99)
    at loadSharedConfigFiles (file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/shared-ini-file-loader/2.2.8/dist-cjs/loadSharedConfigFiles.js:12:76)
    at file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/node-config-provider/2.1.9/dist-cjs/fromSharedConfigFiles.js:8:102
    at file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/property-provider/2.0.17/dist-cjs/chain.js:12:39
    at eventLoopTick (ext:core/01_core.js:183:11)
    at async coalesceProvider (file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/property-provider/2.0.17/dist-cjs/memoize.js:14:24)
    at async file:///tmp/sb-compile-edge-runtime/node_modules/localhost/@smithy/property-provider/2.0.17/dist-cjs/memoize.js:26:28
failed to send request to user worker: connection closed before message completed
InvalidWorkerResponse: user worker failed to respond
    at async Promise.all (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:64:19)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

The two import approaches are loading in different dependencies despite the same version being specified. Interestingly, the same code runs successfully outside of a Deno Deploy function running locally.

@andykais
Copy link

fwiw, I moved over to a deno library for interacting with aws (well, R2 compatible apis in my case) https://deno.land/x/[email protected]. This repo isnt quite as maintained as the official npm sdk, but it works like a charm

@andrefelipe
Copy link

Any lights on this?
A simple S3 delete command is not working on Deno Deploy. It just hangs forever.

@bartlomieju
Copy link
Member

Deno 2.1.9 contains a fix that for several AWS SDK, that was due to missing 100 Continue support (#27381). Gonna close this one as it's quite old and I can't really reproduce the problem no more. Please open a new issue if you're running into any other problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants