Skip to content

Commit

Permalink
Feature: Introduce angular cli and ahead of time compilation (oppia#1…
Browse files Browse the repository at this point in the history
…6073)

* ng-cli

* Successful build

* Angular v14 angular.json

* AOT working, old build not running

* Add webpack module loader

* Fix build after merge from develop

* Fix prod build

* Remove duplicate css and add css styleUrls

* Remove repeated assets.ts code file and fix more css files

* Fix ng build and add start commands

* Add support for production

* Fix logic

* Add more fixes to wrong html files

* Add a mock hashes file to satisfy angular compiler

* New fixes for the template errors

* Bug fixes

* Fix new changes in develop

* Fix new changes in develop

* Review changes

* Fix be tests

* Fix mypy types

* Fix errors

* Lint and backend test fixes

* Fix review comments and add tests

* Merge from upstream/develop

* Fix unit tests

* Fix lint errors

* Fix tests

* Fix tests

* Add a deafult lang direction when no lang dir is present in the url

* Fix build errors in latest develop

* Move document attribution lang setting logic outside the if case

* Fix backend tests

* Upgrade angular and ts version

* yarn lock changes

* Fix backend tests

* Partially finished backend tests

* Downgrade angular back to 11 and move refactored code to ignore list in ts checks

* Fix develop

* Add files to deny lists

* Fix tests

* Fix tests again

* Fix lint

* Fix lint errors

* Fix backend tests

* Fix errors in ng build prod

* Review comments

* Fix upstream/develop

* Addressing review comments

* Fix typings

* Add dist to copied files in e2e tests

* Fix template urls

* Fix multiple browser module imports

* Fix backend tests

* Fix backend tests

* Fix backemd tests

* Fix comments

* Address review comments

* Fix lint and build issues

* Address review comments

* Add more comments and fix lint

* Add comments for declare global

* Re-word the comments

* Fix lint issues

* feconf.py changes for testing aot on backup server

* Revert "feconf.py changes for testing aot on backup server"

This reverts commit fbaa34d.

* Fix review comments and infinite loop reload

* fix hashes

* Change import statement to be relative

* Increase budgets

* Optimize prev lang direction

* Add hashes build step for typescript checks and run_frontend_tests and fix codewowners

* Fix lint and add types

* Fix lint checks

* Fix unit tests and lint errors

* 🫡

* 🫣

* 🫠

Co-authored-by: Vojtěch Jelínek <[email protected]>
Co-authored-by: Vojtěch Jelínek <[email protected]>
  • Loading branch information
3 people authored Dec 20, 2022
1 parent af8dc51 commit 70d3d05
Show file tree
Hide file tree
Showing 252 changed files with 7,283 additions and 3,155 deletions.
9 changes: 7 additions & 2 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,16 @@
/core/templates/services/request-interceptor.service*.ts @oppia/angular-migration-reviewers
/core/templates/services/UpgradedServices.ts @oppia/angular-migration-reviewers
/core/templates/hybrid-router-module-provider*.ts @oppia/angular-migration-reviewers
/core/templates/utility/hashes.ts @oppia/angular-migration-reviewers
/core/templates/utility/string-utility*.ts @oppia/angular-migration-reviewers
/proxy.conf.json @oppia/angular-migration-reviewers
/angular-template-style-url-replacer.webpack-loader.js @oppia/angular-migration-reviewers
/angular.json @oppia/angular-migration-reviewers
/src @oppia/frontend-infrastructure-reviewers

# TS typing
/typings/ @oppia/data-and-stability-reviewers
/tsconfig.json @oppia/data-and-stability-reviewers
/tsconfig-strict.json @oppia/data-and-stability-reviewers
/tsconfig*.json @oppia/data-and-stability-reviewers
/scripts/typescript_checks*.py @oppia/data-and-stability-reviewers


Expand Down Expand Up @@ -573,6 +577,7 @@
/core/templates/components/ck-editor-helpers/ck-editor-copy-content.service*.ts @oppia/interaction-and-rte-reviewers
/core/templates/components/ck-editor-helpers/ck-editor-copy-toolbar/ck-editor-copy-toolbar.component.html @oppia/interaction-and-rte-reviewers
/core/templates/components/ck-editor-helpers/ck-editor-copy-toolbar/ck-editor-copy-toolbar.component*.ts @oppia/interaction-and-rte-reviewers
/core/templates/components/ck-editor-helpers/ck-editor-copy-toolbar/ck-editor-copy-toolbar.module.ts @oppia/interaction-and-rte-reviewers
/core/templates/services/autoplayed-videos.service*.ts @oppia/interaction-and-rte-reviewers
/core/templates/services/external-save.service*.ts @oppia/angular-migration-reviewers
/core/templates/services/external-rte-save.service*.ts @oppia/angular-migration-reviewers
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/e2e_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
# We avoid using ../ or absolute paths because unzip treats these as
# security issues and will refuse to follow them.
run: |
zip -rqy build_files.zip oppia/third_party oppia_tools oppia/build oppia/webpack_bundles oppia/proto_files oppia/app.yaml oppia/assets/hashes.json oppia/proto_files oppia/extensions/classifiers/proto/* oppia/backend_prod_files
zip -rqy build_files.zip oppia/third_party oppia_tools oppia/build oppia/webpack_bundles oppia/proto_files oppia/app.yaml oppia/assets/hashes.json oppia/proto_files oppia/extensions/classifiers/proto/* oppia/backend_prod_files oppia/dist
working-directory: /home/runner/work/oppia
- name: Upload build files artifact
uses: actions/upload-artifact@v3
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ core/templates/prod/*
webpack_bundles/*
core/tests/.browserstack.env
node_modules/*
# .angular is temp folder used by angular to put logs and other artefacts.
.angular/*
# dist is our angular build folder output.
dist/*
.coverage*
!.coveragerc
coverage.xml
Expand Down
105 changes: 105 additions & 0 deletions angular-template-style-url-replacer.webpack-loader.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
// Copyright 2021 The Oppia Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS-IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
* @fileoverview This is a webpack loader that replaces templateUrl: './html'
* with template: require('./html'). This is needed for our webpack based
* compilation and not the angular compiler. Angular compiler parses the html
* and converts it to js instructions. For the style urls, the angular compiler
* uses styleUrls while webpack uses imports. Hence, currently we put the
* stylesheet as import and as a styleUrl in the component. Once we have moved
* away from separate rtl css files, we will remove the import statements and
* just keep styleUrls. Until then, for webpack, we need remove styleUrls for
* webpack compilation.
*/

/**
* The regexes are trying to find the templateUrl from the component decorator
* Eg:
* @Component({
* selector: 'oppia-base-content',
* templateUrl: './base-content.component.html',
* styleUrls: ['./base-content.component.css']
* })
*
* From the above we need to get './base-content.component.html' and
* ['./base-content.component.css'].
*
* After modifications, it will look like:
* @Component({
* selector: 'oppia-base-content',
* template: require('./base-content.component.html'),
* styleUrls: []
* })
* Templates can be found using the regex:
* templateUrl[any number of spaces]:[any number of spaces]
* [any of '"` that starts a string in javascript]
* [match all characters between the quotes][End quotes '"`]
* [any number of spaces]
* [
* ends with a comma or a closing curly bracket depending or wether there are
* more items in the decorator or not
* ]
*/
const TEMPLATE_URL_REGEX = /templateUrl\s*:(\s*['"`](.*?)['"`]\s*([,}]))/gm;
const STYLES_URL_REGEX = /styleUrls *:(\s*\[[^\]]*?\])/g;
const VALUE_REGEX = /(['`"])((?:[^\\]\\\1|.)*?)\1/g;

/**
* This function is only used for templateUrl modifications. From a string this
* function extracts the first value inside quotes ('"`).
* Example: For a string like: "templateUrl: './base-content.component.html',"
* The VALUE_REGEX will match "'./base-content.component.html'" and the first
* group is the quote ("'") and the second group is
* ""./base-content.component.html"
* @param {string} str
* @returns Relative url
*/
const replaceStringsWithRequiresStatement = (str) => {
return str.replace(VALUE_REGEX, function(_, __, url) {
return "require('" + url + "')";
});
};

// eslint-disable-next-line @typescript-eslint/explicit-module-boundary-types
const loader = (sourceString) => {
// https://webpack.js.org/api/loaders/#thiscacheable
// Cacheable is an interface provided by webpack and is used to speed up
// the build by caching results.
this.cacheable && this.cacheable();
/**
* The templateURL regex will match something like:
* "templateUrl: './base-content.component.html',"
*/
const newSourceString = sourceString.replace(
TEMPLATE_URL_REGEX, (_, url) => {
return 'template:' + replaceStringsWithRequiresStatement(url);
}
).replace(STYLES_URL_REGEX, () => {
/**
* For the style urls, the angular compiler
* uses styleUrls while webpack uses imports. Hence, currently we put the
* stylesheet as import and as a styleUrl in the component. Once we have
* moved away from separate rtl css files, we will remove the import
* statements and just keep styleUrls. Until then, for webpack, we need
* remove styleUrls property for webpack compilation.
*/
return 'styleUrl: []';
});

return newSourceString;
};


module.exports = loader;
133 changes: 133 additions & 0 deletions angular.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
{
"$schema": "./node_modules/@angular/cli/lib/config/schema.json",
"version": 1,
"newProjectRoot": "projects",
"projects": {
"oppia-angular": {
"projectType": "application",
"schematics": {
"@schematics/angular:component": {
"style": "scss"
}
},
"root": "",
"sourceRoot": "src",
"prefix": "app",
"architect": {
"build": {
"builder": "@angular-devkit/build-angular:browser",
"options": {
"allowedCommonJsDependencies": ["path", "stream", "zlib"],
"outputPath": "dist/oppia-angular",
"index": "src/index.html",
"main": "src/main.ts",
"polyfills": "core/templates/Polyfills.ts",
"tsConfig": "tsconfig.json",
"assets": [
"src/favicon.ico",
"src/assets"
],
"styles": [
"core/templates/css/oppia.css",
"core/templates/css/oppia-material.css"
],
"scripts": []
},
"configurations": {
"production": {
"fileReplacements": [
{
"replace": "src/environments/environment.ts",
"with": "src/environments/environment.prod.ts"
}
],
"index": {
"input": "src/index.prod.html",
"output": "index.html"
},
"baseHref": "/dist/oppia-angular-prod/",
"outputPath": "dist/oppia-angular-prod",
"aot": true,
"optimization": true,
"outputHashing": "all",
"sourceMap": false,
"namedChunks": false,
"extractLicenses": true,
"vendorChunk": false,
"buildOptimizer": true,
"budgets": [
{
"type": "initial",
"maximumWarning": "500kb",
"maximumError": "2mb"
},
{
"type": "anyComponentStyle",
"maximumWarning": "6kb",
"maximumError": "28kb"
}
]
}
}
},
"serve": {
"builder": "@angular-devkit/build-angular:dev-server",
"options": {
"browserTarget": "oppia-angular:build"
},
"configurations": {
"production": {
"browserTarget": "oppia-angular:build:production"
}
}
},
"extract-i18n": {
"builder": "@angular-devkit/build-angular:extract-i18n",
"options": {
"browserTarget": "oppia-angular:build"
}
},
"test": {
"builder": "@angular-devkit/build-angular:karma",
"options": {
"main": "src/test.ts",
"polyfills": "core/templates/Polyfills.ts",
"tsConfig": "tsconfig.spec.json",
"karmaConfig": "karma.conf.js",
"assets": [
"src/favicon.ico",
"assets"
],
"styles": [
],
"scripts": []
}
},
"lint": {
"builder": "@angular-devkit/build-angular:tslint",
"options": {
"tsConfig": [
"tsconfig.json"
],
"exclude": [
"**/node_modules/**"
]
}
},
"e2e": {
"builder": "@angular-devkit/build-angular:protractor",
"options": {
"protractorConfig": "e2e/protractor.conf.js",
"devServerTarget": "oppia-angular:serve"
},
"configurations": {
"production": {
"devServerTarget": "oppia-angular:serve:production"
}
}
}
}
}
},
"defaultProject": "oppia-angular"
}
5 changes: 5 additions & 0 deletions app_dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,11 @@ handlers:
static_dir: webpack_bundles
secure: always
expiration: "0"
# /dist is the build output folder for angular.
- url: /dist
static_dir: dist/
secure: always
expiration: "0"
- url: /assets
static_dir: assets
secure: always
Expand Down
27 changes: 22 additions & 5 deletions core/controllers/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,16 +79,26 @@ class ResponseValueDict(TypedDict):


@functools.lru_cache(maxsize=128)
def load_template(filename: str) -> str:
def load_template(
filename: str, *, template_is_aot_compiled: bool
) -> str:
"""Return the HTML file contents at filepath.
Args:
filename: str. Name of the requested HTML file.
template_is_aot_compiled: bool. Used to determine which bundle to use.
Returns:
str. The HTML file content.
"""
filepath = os.path.join(feconf.FRONTEND_TEMPLATES_DIR, filename)
filepath = os.path.join(
(
feconf.FRONTEND_AOT_DIR
if template_is_aot_compiled
else feconf.FRONTEND_TEMPLATES_DIR
),
filename
)
with utils.open_file(filepath, 'r') as f:
html_text = f.read()
return html_text
Expand Down Expand Up @@ -647,7 +657,11 @@ def render_downloadable_file(
super(webapp2.Response, self.response).write(file.getvalue()) # type: ignore[misc] # pylint: disable=bad-super-call

def render_template(
self, filepath: str, iframe_restriction: Optional[str] = 'DENY'
self,
filepath: str,
iframe_restriction: Optional[str] = 'DENY',
*,
template_is_aot_compiled: bool = False
) -> None:
"""Prepares an HTML response to be sent to the client.
Expand All @@ -659,6 +673,8 @@ def render_template(
DENY: Strictly prevents the template to load in an iframe.
SAMEORIGIN: The template can only be displayed in a frame
on the same origin as the page itself.
template_is_aot_compiled: bool. False by default. Use
True when the template is compiled by angular AoT compiler.
Raises:
Exception. Invalid X-Frame-Options.
Expand All @@ -683,8 +699,9 @@ def render_template(

self.response.expires = 'Mon, 01 Jan 1990 00:00:00 GMT'
self.response.pragma = 'no-cache'

self.response.write(load_template(filepath))
self.response.write(load_template(
filepath, template_is_aot_compiled=template_is_aot_compiled
))

def _render_exception_json_or_html(
self, return_type: str, values: ResponseValueDict
Expand Down
6 changes: 5 additions & 1 deletion core/controllers/base_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,11 @@ def test_load_template(self) -> None:
with self.swap(feconf, 'FRONTEND_TEMPLATES_DIR', oppia_root_path):
self.assertIn(
'"Loading | Oppia"',
base.load_template('oppia-root.mainpage.html'))
base.load_template(
'oppia-root.mainpage.html',
template_is_aot_compiled=False
)
)


class UniqueTemplateNamesTests(test_utils.GenericTestBase):
Expand Down
Loading

0 comments on commit 70d3d05

Please sign in to comment.