diff --git a/assets/working-hour-distribution.svg b/assets/working-hour-distribution.svg deleted file mode 100644 index 4064e6a33..000000000 --- a/assets/working-hour-distribution.svg +++ /dev/null @@ -1,230 +0,0 @@ - - - - - - - - EN - 中文 - - diff --git a/docs/OpenDigger_Community_Handbook.md b/docs/OpenDigger_Community_Handbook.md deleted file mode 100644 index 6b11edff6..000000000 --- a/docs/OpenDigger_Community_Handbook.md +++ /dev/null @@ -1,206 +0,0 @@ -# Values - -Help open source projects with open-source data analysis. - -# Roles and Responsibilities - -## Committer - -* Facilitate the creation and advancement of software -* Answer questions about the project -* Report on weekly community progress in the regular meeting -* Develop a roadmap -* Respond to issues on the repository -* Review and merge pull requests -* Regularly check the repository for stale content -* monitor issue tracker and pull requests -* steer the architecture of the software -## Contributor - -* Attend community meetings -* Post ideas or problems for new features as issues -* Respond to issues on the repository -* Review pull requests -## Meeting Host - -* Send a reminder about community meetings -* Prepare weekly community meeting -* Host weekly community meeting -* Make sure meeting minutes are kept -* Collect themes from a week in the OpenDigger community -* The host can be changed without any punishment before 4 pm every Thursday -* The host can be changed by deducting the merits that the hostshould getone hour before the meeting -* Count merits of participants in the weekly meeting - * Format: **[yaml](https://yaml.org/)** - * File name: community_meeting_record (All meeting contents are stored in the file `community_meeting_record.yaml`) - * File path: github-analysis-report/community_meeting_record.yaml - * Content like this - -```yaml -time: XXX.XX.XX -host: GitHub ID -attendee: [GitHub ID1, GitHub ID2, GitHub ID3, GitHub ID4] -agenda: - - content: Topic Description1 - proposer: GitHub ID - participants: [GitHub ID1, GitHub ID2, GitHub ID3] - - content: Topic Description2 - proposer: GitHub ID - participants: [GitHub ID1, GitHub ID2, GitHub ID3] -``` - -## Particapant - -* Post ideas or problems for new features as issues -* Create pull requests for code changes -* Respond to reviews on pull requests -## New Member - -* Attend community meetings -* Try to find ways to help -* Catch up on GitHub and other tools used by OpenDigger -* See how you can align what you do elsewhere with what goes on in OpenDigger -* Ask questions about OpenDigger and what is being worked on -# Community Guidelines - -## Things you should take into Consideration - -**Be considerate** - -Our work will be used by other people, and we in turn will depend on the work of others. Any decision we take will affect users and colleagues, and we should consider them when making decisions. - -**Be welcoming and open-minded** - -Other collaborators may not have the same experience level or background as you, but that doesn't mean they don't have good ideas to contribute. We encourage you to be welcoming to new collaborators and those just getting started. We value, encourage all types of contributions. - -**Be respectful** - -Disagreement is no excuse for poor manners. We work together to resolve conflict, assume good intentions, and do our best to act in an empathic fashion. We don’t allow frustration to turn into a personal attack. A community where people feel uncomfortable or threatened is not a productive one. - -**Be collaborative** - -Collaboration reduces redundancy and improves the quality of our work. Internally and externally, we celebrate good collaboration. We prefer to work transparently and involve interested parties as early as possible. - -**Be inclusive** - -Everyone within the community has the right to speak to any decision taken. You should have the freedom to participate in different decisions and actions taken within the community publically and put forward your perspective. Adding your perspective will result in a more broad outcome towards that decision. Whoever you are, you have a unique perspective and unique skills that you can contribute to a collective. - -**Take responsibility for our words and our actions** - -We can all make mistakes; when we do, we take responsibility for them. If someone has been harmed or offended, we listen carefully and respectfully and work to right the wrong. - -**Ask for help when unsure** - -Nobody is expected to be perfect in this community. Asking questions early avoids many problems later, so questions are encouraged, though they may be directed to the appropriate forum. Those who are asked should be responsive and helpful. - -**Step down considerately** - -When somebody leaves or disengages from the project, we ask that they do so in a way that minimizes disruption to the project. They should tell people they are leaving and take the proper steps to ensure that others can pick up where they left off. - -## What is not Allowed? - -* **Threats of violence**- You may not threaten violence towards others or use the site to organize, promote, or incite acts of real-world violence or terrorism. Think carefully about the words you use, the images you post, and even the software you write, and how they may be interpreted by others. -* **Sexually obscene content -**The use of sexualized language or imagery and unwelcome sexual attention or advances -* **Bullying**- Trolling, insulting/derogatory comments, and personal or political attacks -* **Harassment**- Public or private harassment -* **Invasion of privacy**- Publishing others' private information, such as a physical or electronic address, without explicit permission -* **Misinformation**- Other conduct which could reasonably be considered inappropriate in a professional setting -# Path to Leadership - -## What does leadership mean to the OpenDigger community? - -A leader is someone who has specific decision-making power over some part of a group and thus additional accountability and can help with prioritizing and planning activities in the project. - -## How to become a leader? - -### Committer - -Checklist before becoming a **Committer** - -- [ ] Attends community meetings -- [ ] Submits ten commits successfully -- [ ] Nominated by an original committer and gets more than half of the votes of all committers (One-ticket veto) -### Contributor - -Checklist before becoming a **Contributor** - -- [ ] Attends community meetings -- [ ] Submits a commit successfully -### Meeting Host - -Checklist before becoming a **Meeting Host** - -- [ ] Replies his name first in the issue for the weekly host campaign -### Particapant - -Checklist for becoming a **Particapant** - -- [ ] Opens an issue or comments on an issue -- [ ] Comments on a Pull Request -- [ ] Attends community meetings -### New Member - -Checklist for becoming a **Follower** - -- [ ] Stars or Watches or Forks OpenDigger project - - - -# Merit System Rules - -Merits can be divided into two parts: total merits and available merits. Total merits are the sum of all merits and available merits refer to how many merits can be used to redeem reports or other valuable objects. - -## Weekly Meeting - -|Role|merits| -|:----|:----| -|Meeting participants|1| -|Meeting host|13| -|Topic proposer|3| -|Topic participants|2| - -The meeting host needs to count the merits of every participant and then create a pull request to change the merits. The pull request merged after the second inspection by the committer. - -## Issue - -|Action|Role|merits|Others| -|:----|:----|:----|:----| -|Open an issue and others involved|Proposer|3|Most once| -|Comment on issues|Commenter|3 (Including yourself)|Up to 21 merits per person per day| -|Receive evaluation instructions from others|Proposer|merits corresponding to the instruction|Up to 21 merits each issue| - -## Pull Request - -|Action|Role|merits|Others| -|:----|:----|:----|:----| -|Open a pull request and others involved|Proposer|3|Most once| -|Comment on pull requests|Commenter|3 (Including yourself)|Up to 21 merits per person per day| -|Review others' pull requests|Reviewer|8|| -|A pull request be merged|Proposer|8| | -|Receive evaluation instructions from others|Proposer|merits corresponding to the instruction|Up to 34 merits| - -## Evaluation instruction - -When someone brings an evaluation instruction in a comment, Hypertrons 🤖️ will automatically recognize it. - -| |Instruction|merit|Contributor|Committer| -|:----|:----|:----|:----|:----| -|helpful|/good|3|Yes|Yes| -|nice|/great|5|Yes|Yes| -|awesome|/excellent|8|No|Yes| -|incredible|/fantastic|13|No|Yes| - -The evaluation instruction of the pull request is similar to the issue. - -## merits update - -merits will be automatically updated at 02:00 UTC+8 every day by Hypertrons. - -## Others - -A committer can modify merits manually if there may be some unkind actions such as personal attacks and malicious scoring. - - - -**Any opinions will be welcomed!** - diff --git a/docs/README.md b/docs/README.md index 172ba8b5a..2f1c6a39d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,80 @@ # OpenDigger -Open digger is an open source analysis report project for all open source data initiated by [X-lab](https://x-lab.info), this project aims to combine the wisdom of global developers to jointly analyze and insight into open source related data to help everyone better understand and participate in open source. +[![apache2](https://img.shields.io/badge/license-Apache%202-blue)](LICENSE) [![](https://img.shields.io/badge/Data-OpenDigger-2097FF)](https://github.com/X-lab2017/open-digger) [![Node.js CI](https://github.com/X-lab2017/open-digger/actions/workflows/node_ci.yml/badge.svg?branch=master)](https://github.com/X-lab2017/open-digger/actions/workflows/node_ci.yml) + +OpenDigger is an open source analysis report project for all open source data initiated by [X-lab](https://x-lab.info), this project aims to combine the wisdom of global developers to jointly analyze and insight into open source related data to help everyone better understand and participate in open source. + +## Metrics or Indices Usage + +All implemented metrics are open for anyone to use, you can find the data with following URLs, The root URL of OpenDigger static data is `https://oss.x-lab.info/open-digger/github/` right now since we only have GitHub data for now, just replace the `org/repo` or user `login` to get your data. + +Feel free to use the data to construct your own data application and you can refer OpenDigger as your data source and welcome to use the following badge in your project to show the data source. + +[![](https://img.shields.io/badge/Data-OpenDigger-2097FF)](https://github.com/X-lab2017/open-digger) + +### For repos + +| Type | Name | From | Example | Code | CodePen | +| :--- | :--- | :--- | :------ | :--- | :------ | +| Index | OpenRank | [X-lab](https://blog.frankzhao.cn/how_to_measure_open_source_2/) | [openrank.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/openrank.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/indices.ts#L21) | [Demo](https://codepen.io/frank-zsy/pen/bGjyqQj) | +| Index | Activity | [X-lab](https://blog.frankzhao.cn/how_to_measure_open_source_1/) | [activity.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/activity.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/indices.ts#L109) | [Demo](https://codepen.io/frank-zsy/pen/bGjyqQj) | +| Index | Attention | X-lab | [attention.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/attention.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/indices.ts#L235) | [Demo](https://codepen.io/frank-zsy/pen/MWBdpNg) | +| Metric | Active dates and times | [CHAOSS](https://chaoss.community/metric-activity-dates-and-times/) | [active_dates_and_times.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/active_dates_and_times.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L1050) | [Demo](https://codepen.io/frank-zsy/pen/jOpQdZZ) | +| Metric | Stars | X-lab | [stars.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/stars.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/metrics.ts#L15) | [Demo](https://codepen.io/frank-zsy/pen/MWBdpNg) | +| Metric | Technical fork | [CHAOSS](https://chaoss.community/metric-technical-fork/) | [technical_fork.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/technical_fork.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L12) | [Demo](https://codepen.io/frank-zsy/pen/MWBdpNg) | +| Metric | Participants | X-lab | [participants.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/participants.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/metrics.ts#L89) | [Demo](https://codepen.io/frank-zsy/pen/RwBmpYZ) | +| Metric | New contributors | [CHAOSS](https://chaoss.community/metric-new-contributors/) | [new_contributors.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/new_contributors.json)
[new_contributors_detail.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/new_contributors_detail.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L862) | [Demo](https://codepen.io/frank-zsy/pen/RwBmpYZ) | +| Metric | Inactive contributors | [CHAOSS](https://chaoss.community/metric-inactive-contributors/) | [inactive_contributors.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/inactive_contributors.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L965) | [Demo](https://codepen.io/frank-zsy/pen/RwBmpYZ) | +| Metric | Bus factor | [CHAOSS](https://chaoss.community/metric-bus-factor/) | [bus_factor.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/bus_factor.json)
[bus_factor_detail.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/bus_factor_detail.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L780) | [Demo](https://codepen.io/frank-zsy/pen/bGjyqQj) | +| Metric | Issues new | [CHAOSS](https://chaoss.community/metric-issues-new/) | [issues_new.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/issues_new.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L146) | [Demo](https://codepen.io/frank-zsy/pen/mdjaZMw) | +| Metric | Issues closed | [CHAOSS](https://chaoss.community/metric-issues-closed/) | [issues_new.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/issues_new.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L223) | [Demo](https://codepen.io/frank-zsy/pen/mdjaZMw) | +| Metric | Issue comments | X-lab | [issue_comments.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/issue_comments.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/metrics.ts#L52) | [Demo](https://codepen.io/frank-zsy/pen/mdjaZMw) | +| Metric | Issue response time | [CHAOSS](https://chaoss.community/metric-issue-response-time/) | [issue_response_time.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/issue_response_time.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L413) | [Demo](https://codepen.io/frank-zsy/pen/VwBqwaP) | +| Metric | Issue resolution duration | [CHAOSS](https://chaoss.community/metric-issue-resolution-duration/) | [issue_resolution_duration.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/issue_resolution_duration.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L338) | [Demo](https://codepen.io/frank-zsy/pen/VwBqwaP) | +| Metric | Issue age | [CHAOSS](https://chaoss.community/metric-issue-age/) | [issue_age.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/issue_age.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L492) | [Demo](https://codepen.io/frank-zsy/pen/VwBqwaP) | +| Metric | Code change lines | [CHAOSS](https://chaoss.community/metric-code-changes-lines/) | [code_change_lines_add.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/code_change_lines_add.json)
[code_change_lines_remove.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/code_change_lines_remove.json)
[code_change_lines_sum.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/code_change_lines_sum.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L94) | [Demo](https://codepen.io/frank-zsy/pen/dyjByKL) | +| Metric | Change requests
(Open PR) | [CHAOSS](https://chaoss.community/metric-change-requests/) | [change_requests.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/change_requests.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L697) | [Demo](https://codepen.io/frank-zsy/pen/bGjPGxw) | +| Metric | Change requests accepted
(Merged PR) | [CHAOSS](https://chaoss.community/metric-change-requests-accepted/) | [change_requests_accepted.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/change_requests_accepted.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L497) | [Demo](https://codepen.io/frank-zsy/pen/bGjPGxw) | +| Metric | Change requests reviews | [CHAOSS](https://chaoss.community/metric-change-request-reviews/) | [change_requests_reviews.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/change_requests_reviews.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L734) | [Demo](https://codepen.io/frank-zsy/pen/bGjPGxw) | +| Metric | Change request response time | [CHAOSS](https://chaoss.community/metric-issue-response-time/) | [change_request_response_time.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/change_request_response_time.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L415) | [Demo](https://codepen.io/frank-zsy/pen/VwBqwaP) | +| Metric | Change request resolution duration | [CHAOSS](https://chaoss.community/metric-issue-resolution-duration/) | [change_request_resolution_duration.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/change_request_resolution_duration.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L341) | [Demo](https://codepen.io/frank-zsy/pen/VwBqwaP) | +| Metric | Change request age | [CHAOSS](https://chaoss.community/metric-issue-age/) | [change_request_age.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/change_request_age.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/chaoss.ts#L494) | [Demo](https://codepen.io/frank-zsy/pen/VwBqwaP) | +| Network | Developer network | [X-lab](https://blog.frankzhao.cn/github_activity_with_wpr/) | [developer_network.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/developer_network.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/cron/tasks/network_export.ts#L126) | [Demo](https://codepen.io/frank-zsy/pen/NWBVjpV) | +| Network | Repo network | [X-lab](https://blog.frankzhao.cn/github_activity_with_wpr/) | [repo_network.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/repo_network.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/cron/tasks/network_export.ts#L126) | [Demo](https://codepen.io/frank-zsy/pen/NWBVjpV) | +| Network | Project OpenRank | [X-lab](https://blog.frankzhao.cn/how_to_measure_open_source_3/) | [project_openrank_detail/2022-12.json](https://oss.x-lab.info/open_digger/github/X-lab2017/open-digger/project_openrank_detail/2022-12.json) | | [Demo](https://codepen.io/frank-zsy/pen/abjMXBV) | + +### For users + +| Type | Name | From | Example | Code | CodePen | +| :--- | :--- | :--- | :------ | :--- | :------ | +| Index | OpenRank | [X-lab](https://blog.frankzhao.cn/how_to_measure_open_source_2/) | [openrank.json](https://oss.x-lab.info/open_digger/github/frank-zsy/openrank.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/indices.ts#L59) | [Demo](https://codepen.io/frank-zsy/pen/bGjyqQj) | +| Index | Activity | [X-lab](https://blog.frankzhao.cn/how_to_measure_open_source_1) | [activity.json](https://oss.x-lab.info/open_digger/github/frank-zsy/activity.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/indices.ts#L174) | [Demo](https://codepen.io/frank-zsy/pen/bGjyqQj) | +| Network | Developer network | [X-lab](https://blog.frankzhao.cn/github_activity_with_wpr/) | [developer_network.json](https://oss.x-lab.info/open_digger/github/frank-zsy/developer_network.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/cron/tasks/network_export.ts#L63) | [Demo](https://codepen.io/frank-zsy/pen/NWBVjpV) | +| Network | Repo network | [X-lab](https://blog.frankzhao.cn/github_activity_with_wpr/) | [repo_network.json](https://oss.x-lab.info/open_digger/github/frank-zsy/repo_network.json) | [Link](https://github.com/X-lab2017/open-digger/blob/master/src/cron/tasks/network_export.ts#L63) | [Demo](https://codepen.io/frank-zsy/pen/NWBVjpV) | + +## Users + +For above data, there are some users of OpenDigger right now. + +### Applications + +- [OpenLeaderboard](https://open-leaderboard.x-lab.info/): A leaderboard of GitHub world with labeled data. +- [Hypercrx](https://github.com/hypertrons/hypertrons-crx): A browser plugin helps to look into GitHub users and repos. +- [Mulan Dashboard](http://dataease.nzcer.cn/link/1VxPsUCX): A dashboard for [Mulan community](https://portal.mulanos.cn/). +- [Hacking Force China](https://opensource.win/): A ranking list of Chinese developers on GitHub cooperate with [SegmentFault](https://segmentfault.com/). + +### Open source reports + +- [China Open Source Report 2021](https://kaiyuanshe.cn/document/china-os-report-2021/): Most comprehensive open source report in China by [kaiyuanshe](https://kaiyuanshe.cn/). +- [China Open Source Blue Paper 2022](http://www.copu.org.cn/new/308): A blue paper of Chinese open source development by [COPU](http://www.copu.org.cn/). +- [Big Data Open Source Heat Report](cooperations/big_data_open_source_heat_report/开源大数据热力报告2022.pdf): A heat report of open source projects in big data area. + + +## Events + +OpenDigger community also open to inter-community cooperation events, like contests or hackathons. + +- [PaddlePaddle Hackathon 3rd](https://www.paddlepaddle.org.cn/PaddlePaddleHackathon-2022-6), the hackathon final reports are [here](https://github.com/X-lab2017/open-digger/tree/master/cooperations/paddle_hackathon_3rd). ## Data @@ -8,20 +82,22 @@ Open digger is an open source analysis report project for all open source data i We use [GHArchive](https://www.gharchive.org/) as our data source for GitHub event logs and the data service is provided by [clickhouse](https://clickhouse.tech/) cluster cloud service. For data details, please check the [data](https://github.com/X-lab2017/open-digger/blob/master/docs/data.md) docs. -## Contributing guide +### Labeled Data -Please check the [contributing guide](http://www.x-lab.info/open-digger/#/CONTRIBUTING) first if you want to be part of the report. +We are collecting labeled data for more deeper analysis. You can view the corresponding data in the `labeled_data` folder. For more details, please check [labeled_data](labeled_data/README.md) docs. -## Architect & workflow +### Sample Data Usage -Please check the [architect](https://www.x-lab.info/open-digger/#/architecture) and [workflow](https://www.x-lab.info/open-digger/#/workflow) if you want to better understand the project. +OpenDigger provides ClickHouse sample data and Jupyter notebook image to run OpenDigger in local environment, please refer to [sample data doc](./sample_data/README.md). ## Communication -Welcome to join our Slack workspace by clicking the Slack badge above if you want to communicate with us and learn more about the project. Or join the Wechat Group by scanning following QRCode. +Welcome to join the WeChat group by scanning the QRCode and I will invite you into our WeChat group. -![](./assets/wechat-qrcode.png) +
+ +
## License -We use [Apache-2.0 license](https://github.com/X-lab2017/open-digger/blob/master/LICENSE) for code part and [CC-BY-4.0 license](https://github.com/X-lab2017/open-digger/blob/master/LICENSE-CC-BY) for report part, please make sure abide by the licenses when using the project. +We use [Apache-2.0 license](LICENSE) for code part, please make sure abide by the licenses when using the project. diff --git a/docs/_sidebar.md b/docs/_sidebar.md index d1575d86e..c781c3f0e 100644 --- a/docs/_sidebar.md +++ b/docs/_sidebar.md @@ -1,3 +1,5 @@ - [Contributing Guide](/CONTRIBUTING.md) - [Workflow](/workflow.md) - [Data Description](/data.md) +- Metrics + - [OpenRank](/metrics/openrank.md) diff --git a/docs/assets/architect.uml b/docs/assets/architect.uml deleted file mode 100644 index 2fec7c8a1..000000000 --- a/docs/assets/architect.uml +++ /dev/null @@ -1,40 +0,0 @@ -@startuml - -footer Architect of OpenDigger - -actor Developer as dev - -node GitHub as github { - node "open-digger" as repo { - folder ".github/hypertrons-components" as cwf { - file "Custom workflows" - } - folder "global-study" as sqls { - file "All SQLs" - } - folder "case-study" as sqls { - file "All SQLs" - } - } -} - -node "open-digger-bot" as bot -note right: A GitHub app powered by Hypertrons - -node "data service" as ds - -database database as ch -note right: Clickhouse cluster - -database GHArchive as gha -note right: A public service to archive GitHub logs - -dev -down-> repo: Collaborate -bot -up-> github: "Collaborate & manage" - -bot -down-> ds: SQL query -ds -> ch: "Data import & SQL query" - -gha -up-> ds: GitHub logs - -@enduml diff --git a/docs/assets/architecture.svg b/docs/assets/architecture.svg deleted file mode 100644 index aa91b7cb9..000000000 --- a/docs/assets/architecture.svg +++ /dev/null @@ -1,387 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

-
- - - - - - - - EN - 中文 - -
diff --git a/docs/assets/gharchive_raw_data.png b/docs/assets/gharchive_raw_data.png deleted file mode 100644 index b4a611f95..000000000 Binary files a/docs/assets/gharchive_raw_data.png and /dev/null differ diff --git a/docs/community_meeting_record.yaml b/docs/community_meeting_record.yaml deleted file mode 100644 index 6d46bc9fe..000000000 --- a/docs/community_meeting_record.yaml +++ /dev/null @@ -1,179 +0,0 @@ -- time: 2020.6.24 - host: sunshinemingo - attendee: [sunshinemingo, wengzhenjie, frank-zsy, leofang327, kongzhongsanbu, will-ww, xiaoya-Esther, birdflyi, bifenglin,frank-zsy,zhuxiangning] - agenda: - - content: Feature:add sofastack weekly report - proposer: sunshinemingo - participants: [wengzhenjie,frank-zsy] - - content: Discussion-We need to label issues and prs - proposer: sunshinemingo - participants: [frank-zsy] - - content: Discussion-How to organization our meeting content using GitHub - proposer: sunshinemingo - participants: [] - - content: Summer2021 - proposer: sunshinemingo - participants: [leofang327,kongzhongsanbu] - - content: New arch with Next.js + React SSG - proposer: frank-zsy - participants: [] - - content: Document-Metric documentation on web site - proposer: frank-zsy - participants: [] - - content: Dashboard-Visualization dashboard for data service status - proposer: frank-zsy - participants: [sunshinemingo] -- time: 2020.6.17 - host: sunshinemingo - attendee: [sunshinemingo, gymgym1212, kongzhongsanbu, will-ww, xiaoya-Esther, birdflyi, tedliu1, bifenglin,zhuxiangning] - agenda: - - content: Feature Request-add a weekly case study for SOFAStack&MOSN community - proposer: sunshinemingo - participants: [zhuxiangning] - - content: Metrics - proposer: sunshinemingo - participants: [xiaoya-Esther,will-ww,tedliu1] -- time: 2020.6.10 - host: sunshinemingo - attendee: [sunshinemingo, wengzhenjie, gymgym1212, will-ww, birdflyi, tedliu1, bifenglin, zhuxiangning,mb0ma] - agenda: - - content: Add select data sql function - proposer: sunshinemingo - participants: [] - - content: OSSChat - proposer: sunshinemingo - participants: [gymgym1212] - - content: Summer2021 - proposer: sunshinemingo - participants: [will-ww] - - content: China Community Map - proposer: sunshinemingo - participants: [tedliu1, will-ww] -- time: 2020.6.03 - host: sunshinemingo - attendee: [sunshinemingo, wengzhenjie, frank-zsy, gymgym1212, JingbenShi668, kongzhongsanbu, will-ww, xiaoya-Esther, birdflyi, tedliu1, bifenglin] - agenda: - - content: Progress - proposer: frank-zsy - participants: [] - - content: Time period selector - proposer: frank-zsy - participants: [] - - content: Meeting records - proposer: frank-zsy - participants: [sunshinemingo] - - content: Metrics - proposer: frank-zsy - participants: [tedliu1, frank-zsy, will-ww, xiaoya-Esther] - - content: China Community Map - proposer: tedliu1 - participants: [frank-zsy, will-ww] -- time: 2020.5.27 - host: sunshinemingo - attendee: [sunshinemingo, tedliu1, frank-zsy, xiaoya-Esther, wengzhenjie, will-ww, bifenglin, zhuxiangning, birdflyi, gymgym1212, mb0ma] - agenda: - - content: Merit system - proposer: sunshinemingo - participants: [tedliu1, frank-zsy, gymgym1212, will-ww] - - content: Case study requirements - proposer: frank-zsy - participants: [tedliu1, frank-zsy, xiaoya-Esther, will-ww] -- time: 2020.5.20 - host: sunshinemingo - attendee: [sunshinemingo, xiaoya-Esther, wengzhenjie, will-ww, bifenglin, zhuxiangning, birdflyi, yangming, dongshu2013, JingbenShi668, leofang327, gymgym1212] - agenda: - - content: save as pdf function - proposer: frank-zsy - participants: [sunshinemingo] - - content: Chinese open source projects report - proposer: frank-zsy - participants: [sunshinemingo] - - content: case study issue response & resolve period - proposer: frank-zsy - participants: [sunshinemingo] - - content: OssChat - proposer: sunshinemingo - participants: [gymgym1212] - - content: Summer 2021 - proposer: sunshinemingo - participants: [yangming, leofang327, bifenglin, will-ww] -- time: 2020.5.13 - host: sunshinemingo - attendee: [sunshinemingo, tedliu1, frank-zsy, xiaoya-Esther, wengzhenjie, will-ww, bifenglin, zhuxiangning, birdflyi, yangming, dongshu2013, mb0ma, JingbenShi668] - agenda: - - content: Document project name change - proposer: frank-zsy - participants: [] - - content: Report visual style - proposer: frank-zsy - participants: [] - - content: Logo files added into assets - proposer: frank-zsy - participants: [] - - content: Case study of VSCode - proposer: frank-zsy - participants: [will-ww] - - content: Merit system - proposer: frank-zsy - participants: [will-ww, tedliu1] - - content: Summer 2021 - proposer: sunshinemingo - participants: [will-ww, tedliu1] - - content: Contributors - proposer: tedliu1 - participants: [will-ww, tedliu1, frank-zsy] -- time: 2020.4.29 - host: sunshinemingo - attendee: [sunshinemingo, tedliu1, frank-zsy, JingbenShi668, gymgym1212, xiaoya-Esther, wengzhenjie, will-ww, bifenglin, zhuxiangning, birdflyi, yangming] - agenda: - - content: Logo discussion - proposer: frank-zsy - participants: [tedliu1, will-ww, sunshinemingo, xiaoya-Esther] - - content: Pull request collaboration procedure - proposer: frank-zsy - participants: [hi-rustin] - - content: Online global report - proposer: frank-zsy - participants: [hi-rustin] - - content: Credit system details - proposer: frank-zsy - participants: [gymgym1212] - - content: Meeting time - proposer: frank-zsy - participants: [xiaoya-Esther] -- time: 2020.4.22 - host: sunshinemingo - attendee: [sunshinemingo, tedliu1, frank-zsy, JingbenShi668, gymgym1212, xiaoya-Esther, wengzhenjie, will-ww, bifenglin, zhuxiangning, birdflyi, yangming] - agenda: - - content: Logo discussion - proposer: frank-zsy - participants: [tedliu1, will-ww] - - content: Docker image for sample data - proposer: frank-zsy - participants: [bifenglin, will-ww] - - content: Contribution procedure - proposer: frank-zsy - participants: [] -- time: 2020.4.15 - host: sunshinemingo - attendee: [sunshinemingo, wengzhenjie, gymgym1212, zhuxiangning, AnAsteria, will-ww, xiaoya-Esther, yangming, birdflyi, JingbenShi668, tedliu1] - agenda: - - content: New data description - proposer: frank-zsy - participants: [sunshinemingo] - - content: Design of credit system - proposer: gymgym1212 - participants: [wengzhenjie, sunshinemingo, gymgym1212, xiaoya-Esther, tedliu1] -- time: 2022.9.28 - host: xgdyp - attendee: [] - agenda: - - content: An introduction to OpenDigger - proposer: frank-zsy - participants: [] - - content: Framework and Progress of China Open Source Report 2022 - proposer: xgdyp - participants: [frank-zsy,tedliu1,will-ww] - - content: Design of China Open Source Developer Questionnaire - proposer: xgdyp - participants: [] diff --git a/docs/data.md b/docs/data.md index e7efe68d8..82ed7bc0a 100644 --- a/docs/data.md +++ b/docs/data.md @@ -4,13 +4,11 @@ ### Data Source -The data source comes from [GH Archive](https://www.gharchive.org/) which is a project to record the public GitHub timeline, archive it and make it easily accessible for further analysis. Each archive contains JSON encoded events as reported by the GitHub API. The raw JSON data is showing below. There are 6 important data features in this data, namely `id`, `type`, `actor`, `repo`, `payload`, `created_at`. - -![](./assets/gharchive_raw_data.png) +The data source comes from [GH Archive](https://www.gharchive.org/) which is a project to record the public GitHub timeline event log, archive it and make it easily accessible for further analysis. Each archive contains JSON encoded events as reported by the GitHub API. The raw JSON data is showing below. There are 6 important data features in this data, namely `id`, `type`, `actor`, `repo`, `payload`, `created_at`. ### Database -In order to meet the requirement for high-speed analysis among such big data, we parse the row data into well-defined structure and import it into [ClickHouse](https://clickhouse.tech/) server which is an open source column-oriented database management system capable of real time generation of analytical data reports using SQL queries. The Clickhouse database version is 20.8.7.15 in our server. +In order to meet the requirement for high-speed analysis among such big data, we parse the row data into well-defined structure and import it into [ClickHouse](https://clickhouse.tech/) server which is an open source column-oriented database management system capable of real time generation of analytical data reports using SQL queries. The Clickhouse database version is 22.8 in our server. ### Data Schema in Database @@ -20,15 +18,12 @@ The database table offered by the `Clickhouse` server is showing in [data descri For the detailed documentations for Clickhouse SQL usage, check out the [SQL reference](https://clickhouse.tech/docs/en/). -### Examples +### FAQ + +- Q: Can OpenDigger analysis more detailed data for open source repos, like assignee, reactions, label related actions? -There is an example for query data from the Clickhouse database table. You can find more real examples from study SQL components. +- A: No, since `assgin`, `reaction` and `label` events are not included in GitHub timeline event log, we don't have these data in OpenDigger, but you can use GitHub API to get more detailed data and analysis them yourself. -* Pull Request review comment data from an organization +- Q: Why the exported data is not accurate for my open source repos? -``` -SELECT actor_id, actor_login, repo_id, repo_name, issue_id, action, created_at -FROM github_log.events -WHERE type='PullRequestReviewCommentEvent' AND repo_name LIKE '{org}/%' -ORDER BY created_at ASC -``` +- A: Since we are using GHAchive and GitHub timeline event log, there are some data lost due to GitHub API or GHArchive service failure, so the exported metrics can be used to observe the trending but the data can not be treated as an accurate result. diff --git a/docs/metrics/openrank.md b/docs/metrics/openrank.md new file mode 100644 index 000000000..dde0df2c7 --- /dev/null +++ b/docs/metrics/openrank.md @@ -0,0 +1,39 @@ +# OpenRank + +![Type](https://img.shields.io/badge/Type-Index-blue) ![From](https://img.shields.io/badge/From-X--lab-blue) ![For](https://img.shields.io/badge/For-Repo/Developer-blue) + +## Definition + +OpenRank is an index introduced by X-lab, the original idea of OpenRank is from Frank, read the [blog](https://blog.frankzhao.cn/how_to_measure_open_source_2) for the detail of this index. + +OpenRank is a downstream index of `activity`, it partially uses `activity` index to construct a collaborative network for all GitHub repos and developers. The network model is: + +![OpenRankUML](https://www.plantuml.com/plantuml/png/SoWkIImgAStDuUBAJInGI4ajIyt9BqWjKgZcKb0eIymfJLMmjLF8AyrDIYtYgeKeAaejo2_EBCalgiIb2c6CZQwk7R86AuN4v9BCiioIIYukXzIy5A3D0000) + +In the implementation of OpenRank, we use `activity` index as relationship weight for developers and repositories, construct the global network for every month and calculate the OpenRank of every node in the network. However, we do not use `square` to calculate the `activity` in OpenRank because `square` is used to bring community size into account, but for a global collaborative network, the community size is already implied in the network structure. + +Different from PageRank, the value of each node does not entirely depend on the network structure, but also partially depends on the value of the node in last month. So for every developer and repository, it will inherit part of its OpenRank value which is also a reflect of long-term value in open source. + +## Code + +We do not open source OpenRank calculation code in OpenDigger since this is a network index and depends on Neo4j database. But we do export the result of each month to ClickHouse server, so you can still access OpenRank index by the [code](https://github.com/X-lab2017/open-digger/blob/master/src/metrics/indices.ts#L21). + +## Parameters + +There are several parameters used in OpenRank algorithm. + +| Parameter Name | Value | Description | Note | +| :------------- | :---- | :---------- | :--- | +| OpenRank default value | 1.0 | The default value of a new node in the network, like new joined developers and new create repositories | | +| Developer retention factor | 0.5 | How much OpenRank value to inherit from last month for developers | We assume developers' value is much more long-term value than repositories', so developers will inherit more value from themselves | +| Repository retention factor | 0.3 | How much OpenRank value to inherit from last month for repositories | | +| OpenRank attenuation factor | 0.85 | How much OpenRank value will be left if a developer or repository has no activity in a specific month | OpenRank value should not be cleared for repositories and developers if they have no activity just in one month | +| OpenRank min value | 0.1 | Clear the OpenRank value if it attenuates to a smaller one than this value | | + +## CodePen demo + + diff --git a/docs/word-cloud.html b/docs/word-cloud.html deleted file mode 100644 index 9d99ddf2d..000000000 --- a/docs/word-cloud.html +++ /dev/null @@ -1,55 +0,0 @@ - - - - - 词云图 - - - - - -
- - - - - diff --git a/docs/workflow.md b/docs/workflow.md index 1d11d57ff..e9e309c56 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -8,7 +8,7 @@ The automatic procedure of this project is executed by `open-digger-bot`(bot) an ### role -Component [role](https://github.com/X-lab2017/open-digger/blob/master/.github/hypertrons.json#L82) is used to define the roles and their authorities in the community. We have `committer`, `replier`, `sql-reviewer` in this project and details about their authorities will be introduced in each component. +Component [role](https://github.com/X-lab2017/open-digger/blob/master/.github/hypertrons.json#L82) is used to define the roles and their authorities in the community. We have `committer`, `replier` in this project and details about their authorities will be introduced in each component. ### label_setup @@ -22,14 +22,10 @@ Component [weekly report](https://github.com/X-lab2017/open-digger/blob/master/. Component [auto update contribution]() will give a statistics of all developers in the project and create a pull request with a new branch to update [CONTRIBUTORS](https://github.com/X-lab2017/open-digger/blob/master/CONTRIBUTORS) file. This file contains all the roles in the community and `contributor`(developers with commit), `participant`(developers participate in discussion) and `follower`(developers who star or fork the project), if an account appears in a prior role then it will not included in other roles. -The sequence of the roles is `committer`, `sql-reviewer`, `replier`, `contributor`, `participant` and `follower`, we use activity score to sort within a role and the score is appended to the account. For activity calculation, please refer to [GitHub Analysis Report 2019](https://github.com/X-lab2017/github-analysis-report-2019). +The sequence of the roles is `committer`, `replier`, `contributor`, `participant` and `follower`, we use activity score to sort within a role and the score is appended to the account. For activity calculation, please refer to [GitHub Analysis Report 2019](https://github.com/X-lab2017/github-analysis-report-2019). ## Issues -### difficulty - -Component [difficulty](https://github.com/X-lab2017/open-digger/blob/master/.github/hypertrons.json#L133) allows `committer` to use `/difficulty n` comment to add `difficulty/n` label to issue or PR to identify its difficulty, the supported `n` now are 1, 2, 3, 5, 8, which is a Fibonacci sequence. - ### auto_label Component [auto label](https://github.com/X-lab2017/open-digger/blob/master/.github/hypertrons.json#L139) allows the bot to add label to issue or PR automatically, the config details are under `label_setup` component in the `keywords` field of each label which supports multiple keyword to match. @@ -51,7 +47,3 @@ Component [approve](https://github.com/X-lab2017/open-digger/blob/master/.github ### auto_merge Component [auto merge](https://github.com/X-lab2017/open-digger/blob/master/.github/hypertrons.json#L129) will check all open PR with `pull/approved` label and merge them automatically, the check interval is 5 minutes for now. - -### pr_uml_renderer - -Component [PR UML renderer]() is used to render UML files in a PR. This project uses PlantUML to draw the diagrams and use Hypertrons to render the files. But it is really hard for `committer` to review the PR if only text version is provided, so this component will render the UML files in the PR and comment back for check. diff --git a/docs/zh-cn/data.md b/docs/zh-cn/data.md index a5789c7c5..e17dc8416 100644 --- a/docs/zh-cn/data.md +++ b/docs/zh-cn/data.md @@ -6,8 +6,6 @@ 该数据来源于 [GH Archive](https://www.gharchive.org/),这是一个用于记录和归档 GitHub 事件日志的项目。归档的数据为 GitHub 事件日志的 JSON 格式数据,主要包含 6 个重要的字段: `id`, `type`, `actor`, `repo`, `payload`, `created_at`。 -![](../assets/gharchive_raw_data.png) - ### 数据库 为了满足在大规模数据上的高速查询的需求,我们将 GitHub 日志数据解析为结构化数据并导入了开源的列存储高性能实时分析数据库 [ClickHouse](https://clickhouse.tech/) 中,目前该项目使用的 Clickhouse 服务器版本为 20.8.7.15。