|Mosharaf Chowdhury||University of Michigan, Ann Arbor|
|Ankur Dave||UC Berkeley|
|Erik Erlandson||Red Hat|
|Joseph Gonzalez||UC Berkeley|
|Mark Hamstra||ClearStory Data|
|Herman van Hovell||Databricks|
|Shane Knapp||UC Berkeley|
|Cody Koeninger||Nexstar Digital|
|Andrew Or||Princeton University|
|Attila Zsolt Piros||Cloudera|
|Charles Reiss||University of Virginia|
|Kousuke Saruta||NTT Data|
|Shivaram Venkataraman||University of Wisconsin, Madison|
|Matei Zaharia||Databricks, Stanford|
To get started contributing to Spark, learn how to contribute – anyone can submit patches, documentation and examples to the project.
The PMC regularly adds new committers from the active contributors, based on their contributions to Spark. The qualifications for new committers include:
The type and level of contributions considered may vary by project area – for example, we greatly encourage contributors who want to work on mainly the documentation, or mainly on platform support for specific OSes, storage systems, etc.
The PMC also adds new PMC members. PMC members are expected to carry out PMC responsibilities as described in Apache Guidance, including helping vote on releases, enforce Apache project trademarks, take responsibility for legal and license issues, and ensure the project follows Apache project mechanics. The PMC periodically adds committers to the PMC who have shown they understand and can help with these activities.
All contributions should be reviewed before merging as described in
Contributing to Spark.
In particular, if you are working on an area of the codebase you are unfamiliar with, look at the
Git history for that code to see who reviewed patches before. You can do this using
git log --format=full <filename>, by examining the “Commit” field to see who committed each patch.
PRs shall not be merged during active, on-topic discussion unless they address issues such as critical security fixes of a public vulnerability. Under extenuating circumstances, PRs may be merged during active, off-topic discussion and the discussion directed to a more appropriate venue. Time should be given prior to merging for those involved with the conversation to explain if they believe they are on-topic.
Lazy consensus requires giving time for discussion to settle while understanding that people may not be working on Spark as their full-time job and may take holidays. It is believed that by doing this, we can limit how often people feel the need to exercise their veto.
All -1s with justification merit discussion. A -1 from a non-committer can be overridden only with input from multiple committers, and suitable time must be offered for any committer to raise concerns. A -1 from a committer who cannot be reached requires a consensus vote of the PMC under ASF voting rules to determine the next steps within the ASF guidelines for code vetoes.
These policies serve to reiterate the core principle that code must not be merged with a pending veto or before a consensus has been reached (lazy or otherwise).
It is the PMC’s hope that vetoes continue to be infrequent, and when they occur, that all parties will take the time to build consensus prior to additional feature work.
Being a committer means exercising your judgement while working in a community of people with diverse views. There is nothing wrong in getting a second (or third or fourth) opinion when you are uncertain. Thank you for your dedication to the Spark project; it is appreciated by the developers and users of Spark.
It is hoped that these guidelines do not slow down development; rather, by removing some of the uncertainty, the goal is to make it easier for us to reach consensus. If you have ideas on how to improve these guidelines or other Spark project operating procedures, you should reach out on the dev@ list to start the discussion.
Changes pushed to the master branch on Apache cannot be removed; that is, we can’t force-push to it. So please don’t add any test commits or anything like that, only real patches.
To use the
merge_spark_pr.py script described below, you
will need to add a git remote called
as well as one called
apache (the default value of
PUSH_REMOTE_NAME environment variable) is the remote used for pushing the squashed commits
apache-github (default value of
PR_REMOTE_NAME) is the remote used for pulling the changes.
By using two separate remotes for these two actions the result of the
merge_spark_pr.py can be tested without pushing it
into the official Spark repo just by specifying your fork in the
After cloning your fork of Spark you already have a remote
origin pointing there. So if correct, your
git remote -v
contains at least these lines:
apache email@example.com:apache/spark.git (fetch) apache firstname.lastname@example.org:apache/spark.git (push) apache-github email@example.com:apache/spark.git (fetch) apache-github firstname.lastname@example.org:apache/spark.git (push) origin email@example.com:[your username]/spark.git (fetch) origin firstname.lastname@example.org:[your username]/spark.git (push)
apache repo, you will need to set up command-line authentication to GitHub. This may
include setting up an SSH key and/or personal access token. See:
To check whether the necessary write access are already granted please visit GitBox.
email@example.com if you have trouble with these steps, or want help doing your first merge.
All merges should be done using the dev/merge_spark_pr.py, which squashes the pull request’s changes into one commit.
The script is fairly self explanatory and walks you through steps and options interactively.
If you want to amend a commit before merging – which should be used for trivial touch-ups –
then simply let the script wait at the point where it asks you if you want to push to Apache.
Then, in a separate window, modify the code and push a commit. Run
git rebase -i HEAD~2 and
“squash” your new commit. Edit the commit message just after to remove your commit message.
You can verify the result is one change with
git log. Then resume the script in the other window.
Also, please remember to set Assignee on JIRAs where applicable when they are resolved. The script can do this automatically in most cases. However where the contributor is not yet a part of the Contributors group for the Spark project in ASF JIRA, it won’t work until they are added. Ask an admin to add the person to Contributors at https://issues.apache.org/jira/plugins/servlet/project-config/SPARK/roles .
Once a PR is merged please leave a comment on the PR stating which branch(es) it has been merged with.
The trade off when backporting is you get to deliver the fix to people running older versions (great!), but you risk introducing new or even worse bugs in maintenance releases (bad!). The decision point is when you have a bug fix and it’s not clear whether it is worth backporting.
I think the following facets are important to consider:
For me, the consequence of these is that we should backport in the following situations:
We tend to avoid backports in the converse situations: