Working behind proxy

[joaonlopes]

Is your feature request related to a problem? Please describe.

We need our continuous integration to upload the openapi yaml to the Redocly API registry, but we always get time out, we think the reason for this timeout is that we are behind an enterprise proxy.

FetchError: request to https://api.redoc.ly/registry failed, reason: connect ETIMEDOUT 34.238.30.35:44 at ClientRequest.<anonymous> (C:\ (...) \AppData\Roaming\npm\node_modules\@redocly\openapi-cli\node_modules\node-fetch\lib\index.js:1491:11) at ClientRequest.emit (node:events:390:28) at TLSSocket.socketErrorListener (node:_http_client:447:9) at TLSSocket.emit (node:events:390:28) at emitErrorNT (node:internal/streams/destroy:157:8) at emitErrorCloseNT (node:internal/streams/destroy:122:3) at processTicksAndRejections (node:internal/process/task_queues:83:21) { type: 'system', errno: 'ETIMEDOUT', code: 'ETIMEDOUT' }

Authorization failed. Please check if you entered a valid API key.

Describe the solution you'd like

We would like openapi cli working behind a proxy server.

Describe alternatives you've considered

We did a quick modification to your code in registry-api.ts to handle the proxy connection:

const HttpsProxyAgent = require('https-proxy-agent');
private async request(path = '', options: RequestInit = {}, region?: Region) {
    const headers = Object.assign({}, options.headers || {}, { 'x-redocly-cli-version': version });
    if (!headers.hasOwnProperty('authorization')) { throw new Error('Unauthorized'); }
const proxyAgent = new HttpsProxyAgent('XXX:XXX/');
    const response = await fetch(`${this.getBaseUrl(region)}${path}`,  Object.assign({}, options, { headers }, { agent: proxyAgent}) );
    if (response.status === 401) { throw new Error('Unauthorized'); }
    if (response.status === 404) {
      const body: RegistryApiTypes.NotFoundProblemResponse = await response.json();
      throw new Error(body.code);
    }
    return response;
  }

(see: https://www.scrapingbee.com/blog/proxy-node-fetch/)

This seems to work for the login command but still timesout for the push command:

FetchError: request to (...) host failed, reason: connect ETIMEDOUT 52.216.108.107:443 at ClientRequest.<anonymous> ( (...) \openapi-cli\node_modules\node-fetch\lib\index.js:1491:11) at ClientRequest.emit (node:events:390:28) at ClientRequest.emit (node:domain:475:12) at TLSSocket.socketErrorListener (node:_http_client:447:9) at TLSSocket.emit (node:events:390:28) at TLSSocket.emit (node:domain:475:12) at emitErrorNT (node:internal/streams/destroy:157:8) at emitErrorCloseNT (node:internal/streams/destroy:122:3) at processTicksAndRejections (node:internal/process/task_queues:83:21) { type: 'system', errno: 'ETIMEDOUT', code: 'ETIMEDOUT' }

Additional context

I don't know if it matters but we do have an enterprise license for Redocly.

[RomanHotsiy]

Hi @joaonlopes,

Thank your for the feature request. Could you also shoot an email to [email protected] so we can track and prioritize it appropriately.

[joaonlopes]

It's done.

Thank you

[joaonlopes]

Hello,

just an update on this. I managed to successfully push the files after modifying your code in push.ts.

const HttpsProxyAgent = require('https-proxy-agent');
function uploadFileToS3(url: string, filePathOrBuffer: string | Buffer) {
  const fileSizeInBytes =
    typeof filePathOrBuffer === 'string'
      ? fs.statSync(filePathOrBuffer).size
      : filePathOrBuffer.byteLength;
  let readStream =
    typeof filePathOrBuffer === 'string' ? fs.createReadStream(filePathOrBuffer) : filePathOrBuffer;

	const proxyAgent = new HttpsProxyAgent('XXX:XXX/');

  return fetch(url, {
	agent: proxyAgent,
    method: 'PUT',
    headers: {
      'Content-Length': fileSizeInBytes.toString(),
    },
    body: readStream,
  });
}

Hope this helps.

Thank you

[RomanHotsiy]

Thanks @joaonlopes! This definitely helps!

[tatomyr]

Thanks @joaonlopes, I'll prepare the pull request.

[tatomyr]

Hi @joaonlopes,

We introduced the changes in openapi-cli version 1.0.0-beta.81 but have to revert them in 1.0.0-beta.82 due to related issues.
We will come back to this enhancement later. Meanwhile, please use your local fix. Sorry for the inconvenience.

[joaonlopes]

Hello @tatomyr,

thank you very much for the notice. For us version 1.0.0-beta.81 works fine.

The commands preview-docs, linter and push are working fine so we can use that for the time being.

[tatomyr]

Sure @joaonlopes,
Thanks for the update. Glad to hear it works for you.
Do you use the REDOCLY_PROXY env variable for working out your proxy? Just curious if it works for you as we expected.

[joaonlopes]

We just use the REDOCLY_PROXY environment variable and it works fine.

[lornajane]

Can we add a note on what the related issues were that caused this to be reverted? The extra information will help us to implement.

[RomanHotsiy]

Here is the corresponding issue: #555

[joaonlopes]

Hello everyone,

thank you very much for the support you provided for this issue. We are still using beta 81 as part of our deployment, because that version works behind the proxy.

Since you decided to rollback, do you plan on implementing/fixing this? Is it ok to continue to work with beta 81?

Thank you very much

Joao

[RomanHotsiy]

We plan to fix it. We were trying to remember what was the issue that made us rollback.
We'll be updating any progress here.

It's okay to use 81 for now.

[tatomyr]

Tangentially related: #1146.
It would be good to check it when resolving the current one.

[joaonlopes]

Hello everyone,

we are stuck in version 1.0.0-beta.81 and this is starting to cause issues on our side.

For example when using AllOf we are getting a all bunch of validation warnings that don't appear in the latest version:

  _Example value must conform to the schema: `Person` property must NOT have additional properties `DeliveryProof`._

Our problem is that we are working behind a proxy. Is there a workaround for when we want to publish the documentation and we are behind a proxy? I can't believe we are the only users in this situation.

Thanks

[joaonlopes]

I created a fork (https://github.com/joaonlopes/redocly-cli-behind-proxy) with my hack to enable using the CLI behind a proxy. If somebody is having the same difficulties as I do you can use it. It's very rough solution but it works.

The principle is: install a recent version of the redocly cli and replace with the modified versions of registry-api.ts and push.ts

[tatomyr]

Hi @joaonlopes! Sorry, I somehow missed your messages.
We're going to make some changes related to that part of the functionality, and I'd like to incorporate your changes as well. Not sure how this will go though 😅

[RomanHotsiy]

We had to revert the fix unfortunately as it was causing major issues in our other projects.

We will merge it again soon.

[tatomyr]

I just reminded myself that I forgot to update the status of the issue: the changes have finally been released in v1.16.0.
FYI @joaonlopes

[tatomyr]

@joaonlopes we're going to remove the node-fetch dependency, so we had to make some adjustments to how Redocly CLI works with proxies (here's the corresponding PR). We have released a test version with these changes under the snapshot tag. Could you check if it still works for you? The version is 0.0.0-snapshot.1737556585. Thanks in advance!

cc @DmitryAnansky