Skip to content

Latest commit

 

History

History
251 lines (168 loc) · 9.93 KB

e2e-tests-readme.md

File metadata and controls

251 lines (168 loc) · 9.93 KB

End-to-End tests Readme

Year 2021 update:

Read this: docs/wildcard-dot-localhost.md

And: s/tyd u (for "docker-compose up") in one shell, and s/tyd e (for "run end-to-end tests") in another.

Old docs below, some stuff works the same

To do: Document variations, my.conf: E2ECDN, E2EHTTPS

talkyard.secure=true    RENAME to talkyard.https ?
talkyard.secure=false
talkyard.cdn.origin="https://site-1.localhost"
#talkyard.cdn.origin="..."  (commented out)

We use Webdriver.io: https://webdriver.io/docs/api and Typescript.

And you need Yarn: https://yarnpkg.com/en/docs/install And a compiler so you can build Fibers: (which Webdriver.io v6 sync mode uses)

sudo apt-get install g++ build-essential  # do once only

Run tests like so:

  1. In one shell, start the server and database and everything:

     make up

  1. In yet another shell, run the test code. Do one of these: (if there's an issue with node-fibers — then see the next step below)

     # Runs one test: (with "links" in the test file name)
 s/wdio target/e2e/wdio.conf.js --only links

 # To run all tests:
 s/run-e2e-tests.sh        # in Chrome only
 s/run-e2e-tests.sh --all  # in all browsers

 # If you have configured password for third party stuff, like
 # Google and Facebook OpenAuth test accounts:
 s/wdio target/e2e/wdio.conf.js -3 --secretsPath /your/path/to/e2e-secrets.json

    You can choose to run only files that match some pattern. The following runs all test files matching *link*, namely client/test/e2e/specs/all-links.tests.ts:

     s/wdio target/e2e/wdio.conf.js --only 'links'

    If you get this error: "Cannot find module '.../target/e2e/wdio.conf.js'", then fix that by building the end-to-end test code:

     s/d-gulp build-e2e

  2. If this happens: There is an issue with node-fibers ... and: node_modules/fibers/bin/linux-x64-67-glibc/fibers.node is missing, then, try this:

     sudo chown -R $USER.$USER node_modules/  # just in case
 yarn --force  # rebuilds Fibers for your OS and kernel version
 # In the previous step, Yarn probably deleted the Selenium drivers,
 # so need to download them again:
 s/selenium-install

    That'll recompile the fibers.node binary, needed by wdio-mocha-framework.

  3. Some tests, e.g. chat.2browsers.test.ts, require two browsers. Then use the 2whatever config files, e.g.:

     s/wdio target/e2e/wdio.2chrome.conf.js --only 'chat.2browsers'
 s/wdio target/e2e/wdio.3chrome.conf.js --only 'categories.3browsers'

  4. To pause before, or after, the tests, and look at the resulting html pages, use the flag --db (debug before) and --da (debug afterwards). And specify --nt = no-timeouts, to prevent tests-took-too-long errors. Example:

     s/wdio target/e2e/wdio.2chrome.conf.js --only 'chat.2browsers' --da --nt

Testing OpenAuth login

Add --3 --secretsPath=..path-to-gmail-facebook-passwords-file... to the wdio command or to s/run-e2e-tests.sh, e.g.:

s/run-e2e-tests.sh --3 --secretsPath=../e2e-secrets.json

--3 means test-third-party-things. Don't add the e2e secrets file to source control. Have a look in ../tests/e2e/utils/settings.ts, search for "secrets", to see what should be in that file (i.e. e2e-secrets.json). You'd create that file yourself, and keep it secret, don't share it with anyone. Other people create their own test accounts themselves instead.

Google and Facebook break the tests

Some tests test Google/Gmail and Facebook OpenAuth login. Sometimes Google and Facebook pop up extra dialogs asking you security questions, during the login process, and this breaks the tests. So, if the tests break during Gmail and Facebook login, run them manually in a visible browser (use s/selenium-start instead of s/selenium-start-invisible), and answer Google's and Facebook's extra security questions to make them happy.

Browsers other than Chrome

If you want to use a browser other than Chrome, then see Making *.localhost addresses work. (In Chrome, though, ``*.localhost` addresses seem to work fine by default.)

Invisible (headless) tests

If you don't want browser windows to pop up when running the automatic tests, you can run the tests in invisible windows instead. So that you won't get distracted.

On Linux, do this: (other platforms? no idea)

# Install X Virtual Frame Buffer. It can create invisible windows.
sudo apt-get install xvfb  # do once only.

# Start Selenium, via Xvbf, so the browsers Selenium will spawn, will be invisible.
# (Also, specify a not-super-small screen size, otherwise tests will fail.)
xvfb-run -s '-screen 0 1280x1024x8' s/selenium-start

# There's a script for that:
s/selenium-start-invisible

# Run some tests, e.g.:
s/wdio target/e2e/wdio.conf.js --only create-site-admin-guide
s/run-e2e-tests.sh --all  #  -3 --secretsPath /your/path/to/e2e-secrets.json

Flappy tests [flappy_tests]

When running e2e tests, there're many things happening in parallel (browser threads, the browser layout engine and redraws, HTTP requests, server request processing, server background threads, Webdriver.io threads, Webdriver.io HTTP requests), and there can be race conditions that make tests fail, but fail only infrequently so it's hard to reproduce and fix. It's pretty pointless to try to elliminate all such races; instead, we retry any failed test a few times. Only if a test fails somehwat often, its flappiness becomes a problem.

But there's one source of flappy tests that can be elliminated. Namely fetching things from the databse, in undefined order. [flappy_order] Typically PostgreSQL gives us the stuff we're querying for, in always the same order — except for sometimes, and then a test that compares e.g. actual assignees, with expected assignees, could fail. (Or it would have to be written to work with any order.) Therefore, it's good to include order by in, hmm, all? or at least most database queries that return many things (even if we're loading all the things).

Typescript

The tests are written in Typescript. When you run docker-compose start (see above), a Docker container that runs gulp watch (which transpiles Typescript to Javascript) starts. If you want to transpile typescript manually, instead of via the container, then open yet another shell and:

    docker-compose stop gulp
    gulp watch  # when it says "Finished 'build-e2e'", the e2e code has been transpiled

Debugging

For Node.js 7.4:

Don't install node-inspector; it doesn't work with Node 7.4 (compilation errors). Instead, use s/wdio-debug-9101 instead of s/wdio, and add the --db (debug before) param, e.g.:

s/wdio-debug-9101 target/e2e/wdio.2chrome.conf.js --deleteOldSite --db \
    --only new-member-allow-approve.2browsers

The --db flag tells wdio to make a child process it forks debug-listen, and it'll listen on port 9101. In another Bash shell, connect to this child process:

node debug 127.0.0.1:9101

And set breakpoints: setBreakpoint('filename.js', line-number); — however, typing/copy-pasting the correct file name, and finding the Javascript line number (after transpilation from Typescript), takes rather long. So instead I just add debugger; statements in the Typescript source code files. (Don't forget to remove them before committing.)

Unfortunately, currently Chromoe Dev Tools doesn't seem to work with Webdriver.io + Node 7.4. (If you connect Dev Tools (there's a message about how, when you run wdio-debug-9101), you'll connect to the wrong process: the main process, not the forked child process.)

Old, for Node.js < version 7

Don't use >= 6.4, because there's some Node.js/whatever bug that breaks node-inspector.

To debug in node.js:

npm install -g node-inspector  # do once only

# This'll tell you to open a http://127.0.0.1:8080/... in a browser — that's the debugger GUI.
node-inspector --debug-port 5859 --no-preload

# Run the tests, but add --debug = -d, e.g.:
s/wdio target/e2e/wdio.3chrome.conf.js -d --only 'categories.3browsers'

Bug

[E2EBUG] [EVRYBUG] Happened in: tests/e2e/specs/private-chat.3browsers.test.ts (search for "//everyone")

//everyone.waitForAtLeast(1, 'body');  // this, then refresh(), then getHTML('body' or whatever)
                     —>  "Error: stale element reference: element is not attached to the page document"
//browser.waitUntil(() => true);   // ok
//browser.elements('body')  // <— apparently this is makes refresh break  (if everyone === browser)