Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: handle strings the same in cjs, esm, and deno #139

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

isaacs
Copy link

@isaacs isaacs commented Apr 29, 2023

This also ports some of the // istanbul ignore comments to their
associated /* c8 ignore start/stop */ equivalents, and
coverage-ignores some value fallbacks that are there for safety but can
never be hit in normal usage.

Fix: #138

@isaacs isaacs changed the title Isaacs/esm cjs consistency fix: handle strings the same in cjs, esm, and deno Apr 29, 2023
@isaacs
Copy link
Author

isaacs commented Apr 29, 2023

It seems a bit weird to double up the dependencies, but I guess if those modules were shipped as hybrid esm/cjs, it'd be the same amount of code anyway. The only hazard is that the esm versions will probably get bugfixes that aren't going to be backported, but all three of them seem extremely stable, so I think the risk is low.

@isaacs
Copy link
Author

isaacs commented Apr 30, 2023

Is nyc used when testing this library? I left the Istanbul comments in, but maybe they should be removed?

@isaacs
Copy link
Author

isaacs commented Apr 30, 2023

Failure is from the standardx linter, will fix shortly.

@shadowspawn
Copy link
Member

Switched from nyc to c8 in #80, so I think the comments can go.

@isaacs
Copy link
Author

isaacs commented Apr 30, 2023

Hm, the standardx complaint is kind of confusing. It seems like it doesn't like import() statements, but they're clearly valid and supported. I tried declaring it as a global, but doesn't seem to have any effect. Any ideas? I'm not very familiar with standard or eslint, been all in on prettier in my projects for a long time now.

@shadowspawn
Copy link
Member

shadowspawn commented Apr 30, 2023

You have the file named for explicit CommonJS (.cjs): cjs-esm-compare.cjs

Given you want to get access to both implementations and using require+import, should that be .mjs?

Update: well, tried that and didn't help!

@shadowspawn
Copy link
Member

Hacking about without a deep understanding yet, but got new test running. The esm tests are a bit loose as not covered by standardx yet.

Renamed to cjs-esm-compare.mjs and moved down into esm folder. Added to test:esm.

    "test:esm": "c8 mocha ./test/esm/*.mjs",

Reworked to use imports consistently:

'use strict'

/* global describe, it */
import cliuiESM from '../../index.mjs';
import cliuiCJS from '../../build/index.cjs'
import { expect } from 'chai';

const text = `usage: git tag [-a | -s | -u <key-id>] [-f] [-m <msg> | -F <file>] [-e]
               <tagname> [<commit> | <object>]
   or: git tag -d <tagname>...
   or: git tag [-n[<num>]] -l [--contains <commit>] [--no-contains <commit>]
               [--points-at <object>] [--column[=<options>] | --no-column]
               [--create-reflog] [--sort=<key>] [--format=<format>]
               [--merged <commit>] [--no-merged <commit>] [<pattern>...]
   or: git tag -v [--format=<format>] <tagname>...`

describe('consistent wrapping', () => {
  it('should produce matching output in cjs and esm', () => {
    const uiCJS = cliuiCJS({})
    const uiESM = cliuiESM({})
    uiCJS.div({
      padding: [0, 0, 0, 0],
      text,
    })
    uiESM.div({
      padding: [0, 0, 0, 0],
      text,
    })
    expect(uiCJS.toString()).to.equal(uiESM.toString())
  })
})

Then npm run test:esm runs the new test and it passes, but the old test fails (which might be expected since wrapping has changed). Running the test on its own:

% npx mocha test/esm/cjs-esm-compare.mjs


  consistent wrapping
    ✔ should produce matching output in cjs and esm


  1 passing (6ms)

@isaacs
Copy link
Author

isaacs commented May 1, 2023

You have the file named for explicit CommonJS (.cjs): cjs-esm-compare.cjs

Yes, dynamic import() is now CommonJS node modules load ESM modules.

I suppose it could be refactored to be a .mjs test, and then load the cjs one with createRequire().

@isaacs
Copy link
Author

isaacs commented May 1, 2023

Ah, didn't notice test:esm.

Updated the existing esm test to verify the correct behavior, instead of verifying the incorrect behavior. Should be passing now, everything looks good locally.

- Use trimStart/trimEnd instead of deprecated trimLeft/trimRight
This also ports some of the `// istanbul ignore` comments to their
associated `/* c8 ignore start/stop */` equivalents, and
coverage-ignores some value fallbacks that are there for safety but can
never be hit in normal usage.

Fix: yargs#138
@isaacs isaacs force-pushed the isaacs/esm-cjs-consistency branch from a7db817 to 8ee2fad Compare May 1, 2023 04:11
Previously, it was verifying *incorrect* behavior.
@isaacs isaacs force-pushed the isaacs/esm-cjs-consistency branch from 8ee2fad to f236663 Compare May 1, 2023 04:12
@isaacs
Copy link
Author

isaacs commented May 1, 2023

Ah, needed to update deno test as well.

@shadowspawn
Copy link
Member

shadowspawn commented May 1, 2023

For interest I had a look at the impact in package size. As might be expected, installed size about doubles since two of everything. 😅

Details
# OLD

% npm pack                                        # master

npm notice 📦  [email protected]
npm notice 
npm notice 📦  [email protected]
npm notice === Tarball Contents === 
npm notice 731B   LICENSE.txt              
npm notice 3.0kB  README.md                
npm notice 10.0kB build/index.cjs          
npm notice 1.1kB  build/index.d.cts        
npm notice 9.7kB  build/lib/index.js       
npm notice 1.0kB  build/lib/string-utils.js
npm notice 309B   index.mjs                
npm notice 2.0kB  package.json             
npm notice === Tarball Details === 
npm notice name:          cliui                                   
npm notice version:       8.0.1                                   
npm notice filename:      cliui-8.0.1.tgz                         
npm notice package size:  6.4 kB                                  
npm notice unpacked size: 27.7 kB                                 
npm notice shasum:        b1a4ebefef09e89647b03e97b38bb5ad51ad77ba
npm notice integrity:     sha512-54py6xQmmLGZW[...]wTZmWq1VmXvRw==
npm notice total files:   8                                       
npm notice 

% npm i /Users/john/Documents/Sandpits/yargs/cliui/my-fork/cliui-8.0.1.tgz
added 10 packages, and audited 11 packages in 1s

% du -d=1 -h node_modules 
312K	node_modules

% npm ls --all
[email protected] /Users/john/Documents/Sandpits/play/try-cliui/old
└─┬ [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ └── [email protected] deduped
  ├─┬ [email protected]
  │ └── [email protected]
  └─┬ [email protected]
    ├─┬ [email protected]
    │ └─┬ [email protected]
    │   └── [email protected]
    ├── [email protected] deduped
    └── [email protected] deduped
# NEW

% npm pack                                         # isaacs-isaacs/esm-cjs-consistency
npm notice 
npm notice 📦  [email protected]
npm notice === Tarball Contents === 
npm notice 731B   LICENSE.txt       
npm notice 3.0kB  README.md         
npm notice 10.4kB build/index.cjs   
npm notice 1.1kB  build/index.d.cts 
npm notice 10.1kB build/lib/index.js
npm notice 299B   index.mjs         
npm notice 2.2kB  package.json      
npm notice === Tarball Details === 
npm notice name:          cliui                                   
npm notice version:       8.0.1                                   
npm notice filename:      cliui-8.0.1.tgz                         
npm notice package size:  6.1 kB                                  
npm notice unpacked size: 27.7 kB                                 
npm notice shasum:        20504c591d90743add1b812cf439b9d845108468
npm notice integrity:     sha512-F9qOs1ByD8UAg[...]uVxRAh/gdtH3g==
npm notice total files:   7                                       
npm notice 

% npm i /Users/john/Documents/Sandpits/yargs/cliui/my-fork/cliui-8.0.1.tgz
added 23 packages, and audited 24 packages in 5s

% du -d=1 -h node_modules 
756K	node_modules

% npm ls --all
[email protected] /Users/john/Documents/Sandpits/play/try-cliui/new
└─┬ [email protected]
  ├─┬ string-width-cjs@npm:[email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ └─┬ [email protected]
  │   └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ └── [email protected] deduped
  ├─┬ strip-ansi-cjs@npm:[email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├─┬ wrap-ansi-cjs@npm:[email protected]
  │ ├─┬ [email protected]
  │ │ └─┬ [email protected]
  │ │   └── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ ├── [email protected] deduped
  │ │ └── [email protected] deduped
  │ └─┬ [email protected]
  │   └── [email protected]
  └─┬ [email protected]
    ├── [email protected]
    ├── [email protected] deduped
    └── [email protected] deduped

isaacs added a commit to isaacs/jackspeak that referenced this pull request May 1, 2023
@isaacs
Copy link
Author

isaacs commented May 2, 2023

Anything else needing to be done to land this? Some folks getting upset at me using a git dep while waiting.

@isaacs
Copy link
Author

isaacs commented May 2, 2023

(Apparently the build fails when installing learna for some reason.)

@shadowspawn
Copy link
Member

shadowspawn commented May 2, 2023

Anything else needing to be done to land this?

Nothing else from you at this point from me anyway, thanks.

I'll review it as a supply-both approach to getting wrapping working in esm et al.

A high level question is whether we want to land a double-dependency update, and I'll want input from @bcoe on that. I know people have expressed concerns about number of dependencies and size of Yargs install in past so this is of some concern, which is why I checked install sizes for reference. (And Benjamin has also mentioned heading for ESM whether esm-first or esm-only at some point.)

Some folks getting upset at me using a git dep while waiting.

I don't have a timeline for you. Could you work around the limitation in the meantime by using require from esm and getting the CommonJs implementation? (I am not sure whether this makes any sense for your runtime setup.)

@isaacs
Copy link
Author

isaacs commented May 2, 2023

I'll just publish my fork as @isaacs/cliui for now. Was just being lazy. I don't mind littering my npm account, it's a garbage pile of throwaway junk after all these years anyway lol

@OpportunityLiu
Copy link

The "string-width-cjs": "npm:string-width@^4.2.0" part breaks in yarn v1. While other packages still deps "string-width@^4", yarn install only one string-width-cjs in root of node_modules.

"string-width-cjs@npm:string-width@^4.2.0", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.3:
  version "4.2.3"
  resolved "https://registry.npmmirror.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010"
  integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==
  dependencies:
    emoji-regex "^8.0.0"
    is-fullwidth-code-point "^3.0.0"
    strip-ansi "^6.0.1"
> yarn why string-width    
yarn why v1.22.19
[1/4] Why do we have the module "string-width"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "[email protected]"
info Reasons this module exists
   - "web-ext#update-notifier#boxen" depends on it
   - Hoisted from "web-ext#update-notifier#boxen#string-width"
   - Hoisted from "web-ext#update-notifier#boxen#widest-line#string-width"
   - Hoisted from "web-ext#update-notifier#boxen#wrap-ansi#string-width"
   - Hoisted from "rimraf#glob#jackspeak#@isaacs#cliui#string-width"
info Disk size without dependencies: "200KB"
info Disk size with unique dependencies: "280KB"
info Disk size with transitive dependencies: "300KB"
info Number of shared dependencies: 4
Done in 0.54s.
> yarn why string-width-cjs
yarn why v1.22.19
[1/4] Why do we have the module "string-width-cjs"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "[email protected]"
info Reasons this module exists
   - "yargs" depends on it
   - Hoisted from "yargs#string-width-cjs"
   - Hoisted from "yargs#cliui#string-width-cjs"
   - Hoisted from "web-ext#yargs#string-width-cjs"
   - Hoisted from "yargs#cliui#wrap-ansi-cjs#string-width-cjs"
   - Hoisted from "web-ext#addons-linter#yargs#string-width-cjs"
   - Hoisted from "rimraf#glob#jackspeak#@isaacs#cliui#string-width-cjs"
   - Hoisted from "web-ext#update-notifier#boxen#ansi-align#string-width-cjs"
info Disk size without dependencies: "84KB"
info Disk size with unique dependencies: "164KB"
info Disk size with transitive dependencies: "184KB"
info Number of shared dependencies: 4
Done in 0.55s.

@shadowspawn
Copy link
Member

@isaacs Does using a CommonJS implementation in ESM work for clients of jackspeak?

You mentioned createRequire in #138 (comment) so might be ok, but you may have decided since then that really want "true ESM"!

Related: In older discussion, Deno and Yargs browser support were the scenarios of concern for "true ESM": #89 (comment)

@isaacs
Copy link
Author

isaacs commented May 8, 2023

"True ESM" is definitely the goal in principle, as eventually I would like tap to work in deno and browsers, so I've been shipping full hybrid builds for everything. That said, the stuff using cliui won't ever work in browsers per se, and most of the cli runner is pretty node specific, so getting to deno support will be a big lift as well, so just shipping CJS is fine, too. The main thing for me here is, import * from cliui should be functionally equivalent to require('cliui'). However you wanna accomplish that is fine.

@shadowspawn
Copy link
Member

Good info, thanks @isaacs

The reason I am considering different approaches is that the underling lack of ansi support in ESM flavour of cliui generated a few issues across yargs in a couple of years. Using the npm aliases via jackspeak generated a couple of reports within days (due to breaking builds). Which we only know now because you tried an approach @isaacs , full credit for that!

@naugtur
Copy link

naugtur commented Jul 28, 2023

When yarn3 attempts to run audit on a project with @isaacs/cliui in dependencies, it gets a HTTP400 from the audit API because of the *-cjs references:

https://github.com/isaacs/cliui/blame/main/package.json#L53

Repro with details:
danjm/yarn-audit-bug#1

@shadowspawn
Copy link
Member

I wasn't able to reproduce this. I cloned out your repo, and ran (on Mac):

% yarn --version
3.2.4
% yarn install
...
% yarn npm audit
➤ YN0001: No audit suggestions

@shadowspawn

This comment was marked as outdated.

@shadowspawn
Copy link
Member

I pulled from correct repo, but still can not reproduce. What commands are you running that are showing an issue?

@naugtur
Copy link

naugtur commented Jul 28, 2023

Sorry!
You need to add --recursive because otherwise yarn won't send transitive dependencies. I'll update the description there.

@isaacs
Copy link
Author

isaacs commented Jul 28, 2023

...?!

Yarn doesn't audit transitive deps by default??

@naugtur
Copy link

naugtur commented Jul 28, 2023

🤷‍♂️ c'est la vie

@shadowspawn
Copy link
Member

Ok, reproduced now.

I have an alternative PR for cliui prepared, but don't have any yarns knowledge to try plugging it in to your repo and see if it performs better. If you can easily try this repo as a replacement for the code from this PR, I would love to know if avoids the problem. No worries if you are not interested!

@Bonnie-dot
Copy link

I meet the same issue with yarn. I went through all comments, but I didn't find any solution except using npm. And I used npm, it can works. Hopefully it can be solved in the future.

@G-Rath
Copy link

G-Rath commented Sep 16, 2023

@shadowspawn I've tried your PR and confirmed it doesn't have the same problem as @isaacs/cliui 🎉

@LennonReid
Copy link

LennonReid commented Sep 19, 2023

I encountered the same issue since yesterday. However, after switching to PNPM, the problem was resolved. Running the command 'yarn why cliui' pointed me to the @ngneat/transloco. Thanks for the guidance; you all made my day!

@shadowspawn
Copy link
Member

For yarn 1 users hitting this page because of runtime failures with cliui and string-width, try upgrading to yarn v1.22.22 or higher.

shadowspawn
shadowspawn previously approved these changes Mar 9, 2024
Copy link
Member

@shadowspawn shadowspawn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a yarn 1 update available, I think this is now ok.

I have an alternative PR open which uses rollup plugins rather than a package manager alias to support the cjs/esm implementations, different tradeoffs but similar outcome: #143.

happyit1212 added a commit to happyit1212/CoolcoderJacker that referenced this pull request Jun 11, 2024
@shadowspawn shadowspawn dismissed their stale review July 6, 2024 23:13

I do prefer approach in #143 and still getting people reporting issues from the breakage in yarn 1

Comment on lines +52 to +57
"string-width": "^5.1.2",
"string-width-cjs": "npm:string-width@^4.2.0",
"strip-ansi": "^7.0.1",
"strip-ansi-cjs": "npm:strip-ansi@^6.0.1",
"wrap-ansi": "^8.1.0",
"wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0"
Copy link

@slorber slorber Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how helpful this is, but this change in @isaacs/cliui fork scared me.

I'm running this on the Docusaurus repo, trying to detect possible supply chain attacks, and got these aliases being reported:

npx lockfile-lint --path yarn.lock --type yarn --allowed-hosts yarn --validate-https --validate-package-names
detected resolved URL for package with a different name: string-width-cjs
    expected: string-width-cjs
    actual: string-width

detected resolved URL for package with a different name: strip-ansi-cjs
    expected: strip-ansi-cjs
    actual: strip-ansi

detected resolved URL for package with a different name: wrap-ansi-cjs
    expected: wrap-ansi-cjs
    actual: wrap-ansi

 ✖ Error: security issues detected!

And there's this "anonymous" guy that published empty packages on npm with the exact same name:
https://www.npmjs.com/package/string-width-cjs
https://www.npmjs.com/package/strip-ansi-cjs
https://www.npmjs.com/package/wrap-ansi-cjs

I don't know how harmful it could be, but this looks suspicious that those packages even get a few weekly downloads.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a security issue, it's just a dependency alias, which every js package manager should support by now. If yarn is complaining, it's likely a yarn bug.

Those packages likely get downloads because the registry is constantly being mirrored by many third-party registry instances. I don't know if they're malicious or just litter, but they're irrelevant here.

Copy link
Member

@shadowspawn shadowspawn Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That a might be a false positive from lockfile-lint, possibly it does not look for package aliases. The author of lockfile-lint may be interested.

If is interesting that there are some matching packages published. The optimistic view is perhaps someone was investigating working around the bug in earlier versions of yarn. The pessimistic view is someone was investigating exploiting the problem. The failures we have seen are a runtime failure rather than a download so not directly fixable/exploitable in this way. (But someone might possibly see the alias name in a message and think it was missing and try installing it.)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is definitely a false positive in whatever tool is generating this warning. I recommend reporting it to them.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isaacs considering that the npm registry itself is/was vulnerable to npm package aliases and allowed spoofing package names (the way you used them here in the package.json) is enough to warrant that this is a real-world concern and not theoretical, in my opinion. See here for evidence: https://snyk.io/blog/exploring-extensions-of-dependency-confusion-attacks-via-npm-package-aliasing/

@slorber lockfile-lint allows you to accept the risk of package aliases as long as you explicitly call them out, here's how to allow-list one of the packages:

npx lockfile-lint --path package-lock.json --allowed-hosts yarn npm --validate-https --validate-package-names --allowed-package-name-aliases string-width-cjs:string-width

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lirantal That blog post is misleading. It's not a "supply chain security" issue, it's just a website display issue. I'm not saying it's not a bug, and yes it does potentially lead to falsely providing reputation in some way to the other package name, but it's imo quite a stretch to call it evidence of a supply chain security bug for package publishers or consumers using package aliases as they're intended. The website is not part of the supply chain, and the registry and all modern clients of it handle aliases just fine.

The failure of lockfile-lint to do so as well leads to false positives like this one creating make-work and wasting everyone's time. Please reconsider.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Wrapping logic seems to be weirdly different between ESM vs CJS versions
9 participants