Changes since 0.20.0
- limit delta's sent to services
- added script to optimize job schedules based on run times
- bumped scraper to 1.3.0:
- cleanup meetingburger dynamic urls
- only store pages with Notulen, Agenda, Besluitenlijst, Uiittreksel, besluit or behandelingvanagendapunt
- include scraping report as output of scraper
Changes since 0.19.0
- bumped validation and diff service to add configurable sleep time
- besluiten export script was improved and uses batching now
- metric added to detect long running jobs
- various fixes to metrics endpoint
- bumped deltanotifier & enabled retry on delta's to job-controller
Changes since 0.18.1
- added metrics endpoint with basic metrics
- bumped sparql parser to 0.0.13
- bumped scraper to 1.2.0 (limits max amount of items scraped)
- bumped harvest import service to 0.13.5
- bumped harvest validator to 0.3.7
- bumped diff service to 0.2.6
- bumped sameas service to 4.5.1
- bumped graph maintainer to 1.3.0 (limits the maximum amount of delta files returned in one call)
- bumped delta-producer-background-jobs-initiator to 1.0.1
- bump dump file publisher
- bump frontend with new features
- add login feature flag (turn login on/off)
- model change, store job auth info in harvest-collection instead of remote data object
- add authentication configuration to harvest-collection model (resources)
- bump harvest collector to support authentication configuration model change
- added support for authentication in normal jobs (scheduled jobs is still WIP)
- improve performance by using a diff step by @nbittich in #19
- Upgrade virtuoso's config to offset to more than 1 million by @claire-lovisa in #17
- Add agendas and decision lists to the export by @claire-lovisa in #14
- Add error-alert service to receive emails when things go wrong by @claire-lovisa in #15
- Skip mu-auth for dump and initial sync jobs by @claire-lovisa in #16
- Add jobs to harvest Cevi and Remmicom by @claire-lovisa in #18
- Bump frontend which includes some new features:
- Allow users to filter the results by status
- Show the target URI in the overview table
- Revert frontend bump - causes issues with previous data
- Sprintf worarounds
- Graph updates
- URI updates
- Bump multiple dependencies to use latest version of mu-auth-sudo (retry mechanism on queries)
- Bump frontend
- Added the possibility to partially update scheduled jobs
- CVE-2021-45046: see https://spring.io/blog/2021/12/10/log4j2-vulnerability-and-spring-boot
- bump of various services related to deltas (fixing bugs mainly)
- Important changes related to performance
- harvest-collector: added queue processing and use parts of comunica instead of marawa to speed up parsing.
- harvesting-import: use parts of comunica instead of marawa to speed up parsing.
- Bump new version of import-sameas, which might tackle some hanging jobs if no triples were found.
- Scheduled jobs
- Less strict validation for motivation
- Update producer for missing properties on besluit and artikel
- Report on broken URl's after publishing