Intro to Git and GitHub

Frank Aragona & Juan Salazar, DIQA

the problem

• multiple files for the same script
• easy to lose track of production version
• terrible for collaboration
  • who has the main script?
  • who deleted my changes?
  • conflicts?
  • I overwrote my code and lost everything

20211221_AllTests_datatable_conversion.R

20211221_AllTests_datatable_conversion_local.R

20211221_AllTests_updated.R

20211223_AllTests_dplyr_FA.R

20211221_AllTests_dplyr_conversion_JS.R

- We’ve probably all been in this scenario where we have multiple versions of the same file, each with slightly different names or dates

- nobody remembers what each file really contains or how they differ or which is actually the most recent,

- and it creates a chaotic environment where we don’t know what the source of *truth* is:

• so we lose track of the most recent production version,

• and it’s terrible for collaboration

  • Which script should we be running?

  • Another user deleted updates I made and now we have conflicting versions of the script on each of our computers

  • Or worst case scenario, you accidentally overwrite everything you worked on

- Teams often try to develop their own standards to control script versions internally, but there is no better management system than Git.

what is Git?

• Git is version control software
• projects have branches
• advantages:
  • create new branches to test code
  • isolate while keeping the main branch clean
  • version-tracking, keeps track of everything
  • prevents overwriting/losing your changes

• What is Git?

• Git is industry-standard version control software. It lets you isolate your *main* code from your development code

• projects have branches; in this figure you can see a *main* branch that holds the *production* or *source of truth* code in blue

• if someone wants to edit or add a new *feature*, they can make a new isolated copy, or branch, of that *main* code (seen here in yellow).

• Major advantages of this:

  • It allows you to create new branches to test code before you implement it.

  • Users can isolate new branches from your main branch to keep the main branch clean and running until you are able to safely launch updates

  • It lets you see the entire history of your code base through really well-documented version-tracking

  • It prevents accidental overwrites of your code

  For example, for the RESP-NET project my team has our main production branch that creates a weekly analytic file which we share with the CDC. However, we are constantly making improvements. If somebody on our team wants to update our code to report a new racial category, like Middle-Eastern or North African (MENA), we don’t want that user to make those changes directly in our *main* code until they have been thoroughly tested. So they would create a new branch off of main, make the necessary updates, and then merge them back into main once they’re ready. Git lets you do this, seamlessly.

But to effectively use git, you also need to use a git management tool. And the most popular tool to use is GitHub.

what is GitHub?

• GitHub is a graphic user interface platform that hosts Git repositories
  • repos are used to store project files
• advantages:
  • code management: isolate your changes from main codebase
  • project management
  • collaboration
  • transparent, helps see everything

• what is GitHub?
• GitHub is a graphic user interface platform that hosts your Git repository, or Git Projects, online
  • It allows people to share, review, and contribute to each other’s projects/repositories
    • a repository, or “repo” is a place to store your project files
    • the figure on the right shows a Git repository or “repo”. Each team member can make a clone, or copy, of the repo and work on local changes to the code
• the main advantages of using GitHub are:
  • it facilitates isolating your specific changes from the main codebase which allows for easier code management
  • It supports project management through issue tracking
  • it’s where you will collaborate and discuss code with your teammates or external partners
  • and it lets you visualize the entire Git history

tl;dr

• Git repos -> movies 🎞️
• GitHub -> Netflix

• Git repositories, or projects, are like movies

• GitHub is like Netflix - it stores all your movies

• Imagine your team wants to edit scenes in a movie

• Git allows your team to make edits to the scenes and keeps track of all the changes that were made

• It also helps to prevent conflicts between your scene edits and your teammates edits.

• When you want to work on a project, you can pull the movie from Netflix and work on scene edits in isolation

• This is an incomplete analogy, however, because GitHub can do so much more

benefits of GitHub - code management

Features:

• version history (commits): easily view the entire version history of your codebase
  • change attribution: know who did what, and why
  • time travel: roll back to previous versions
• structured repositories: organizes scripts and documentation clearly

GitHub helps manage code through transparency and structure, which are extremely important when there are multiple stakeholders contributing to your codebase or when your team is trying to create reproducible data. Some of the features GitHub has to support code management are:

• Version history, which is tracked through commits. Every time a user makes a change, GitHub creates a snapshot of your code so you can go back and see what changed and when. In this screenshot, we are looking at the change between a previous version of the script (on the left) and an updated version (on the right). It highlights exactly what has changed in the file. The user here has replaced the password so that it’s not stored in in their script anymore.

  • version history also enables change attribution. Every commit is tied to a person and a message describing what they did. You can see what user added or deleted code and why they did it. This is really helpful to understand and remember the context behind code edits when you revisit projects weeks or months later.

  • version history also gives you the power of time travel. If something in your codebase were to break, or you needed to undo a change, you can easily roll back to previous versions. Everything is stored in a centralized place, so there’s no need to go digging through your personal files or files on the Y drive.

• Finally, GitHub encourages structured repositories as a means to organize scripts and documentation consistently. It also let you clearly separate production-ready code from work that is still in development. And you can use folders or repos to separate different projects.

These features all help prevent mistakes and help teams work more effectively, especially if code is being reused, shared, or updated over time.

benefits of GitHub - project management

Features:

• issues: track tasks, decisions, bugs, and enhancements
  • labels and assignees: categorize work and assign ownership
  • organization: easy to tag and search tasks
• milestones: group issues to track progress towards goals

Beyond managing code, GitHub also has some tools for managing projects. Some features worth highlighting:

• issues. These are like tasks or to-do items and they can be used to track tasks, document decisions, track bugs in your code, or request enhancements/new features for your codebase.
  • As part of issues, Github has labels and assignees. Labels help categorize work. My team will use these to define enhancements, priority items, documentation fixes, etc. And you can assign issues to specific team members so it’s clear who is responsible for what. In this screenshot, we have a few issues with descriptive titles. In the bubbles to the right, are labels to help categorize issues. At the far right, you can see the user which has been assigned this task.
  • so Issues are very well-organized. You can easily search, filter, or tag issues and you can link issues directly to code changes. For example, if my team needed to fix a bug, we can easily connect that fix to the issue it resolves and create a clear history of how the code was updated.
• Lastly, there’s also the ability to group issues together in milestones so you can track progress toward larger goals, like creating a new report or publishing a new dashboard.

If you’ve used tools ike Smartsheet, GitHub’s project board also offers some similar way to track project milestones there.

Overall, GitHub has some useful tools to facilitate project management. My team heavily relies on issues since they live alongside our codebase and can easily grow with the project without us having to manage a separate tracker or spreadsheet for codebase updates. I’m going to show quick example of an issue that Frank and I used to create this presentation. This is the issues page and we have 2 issues. Issue #2 todo was created by Frank and he assigned it to himself. In here, we’ve been communicating and tracking idea for this presentation. You can see on the right that we’re able to assign other users, create labels, attach the issue to a specific project or milestone. So I find issues extremely useful for assigning projects and keeping track of them.

benefits of GitHub - collaboration

Features

• branches: team members work on different tasks without conflict
• pull requests: enable code review and discussion
  • comment & review: clear contextual communication history
• sharing code: easily share and control access to your code
  • organizations, private/public repos

One of GitHub’s biggest strengths is how it supports collaboration through the entire workflow.

• I’ve already mentioned this, but it’s worth repeating, branches let team members work on different tasks, at the same time, without conflict. I might be updating and reviewing code to include eCR labs in our reporting scripts, while somebody else is working on adding a new QA table to our email reports. We can both work in parallel, while our main codebase remains stable.

• Once a task is ready, we use pull requests to propose the update. This gives teams a chance to review the code, discuss it, and even request edits before anything gets merged into the main codebase.

• Within the pull request, users can comment & review. I use this with my team to ask questions, leave feedback, or tag other collaborators for their input. You’re even able to highlight specific lines of code that you think may need further review. Everything is neatly saved, and organized and easy to revisit later.

  • In this screenshot, a teammate asked me to review their code. I found a small issue, requested a change, and left a comment with my suggested fix. GitHub makes it really simple to track those updates. So now I can see specifically what my team member updated since the previous review - he also left a comment with a description. and it’s really easy to grant final approval once everything’s ready to merge into the main codebase.

• Very importantly, GitHub makes sharing code extremely simple. Admins are able to control who can see, and who can edit code through the use of organizations, public repos, and private repos which I’ll cover a bit more later.

Personally, I’ve found pull requests and code reviews incredibly useful for onboarding new users. They help me track whether all requested features were added, and if someone is coding in a language they’re still learning, I can provide targeted, helpful feedback while documenting their updates in an easily reviewable way.

what do we use it for

• all code development
  • surveillance pipelines
  • single ‘one-off’ scripts
• large collaborative organizations to share code
  • public repos: share code broadly or collaborate with external partners
  • private repos: restricted access for sensitive internal work
  • organizations: WA-EIP, NW-PaGe

How does our team, DIQA, use git and GitHub?

• It is one of our primary tools and we use it for all code development.

• This includes our major surveillance pipelines which extract, transform, and load surveillance data for routine reporting, but also includes simpler “one-off” analysis scripts or troubleshooting code.

• an important aspect of our work is cross-team collaboration, especialy with other epi teams and program parners. To support that, we’ve created large collaborative GitHub organizations to share our code and contribute to projects in a centralized location.

  • In Github, an organization is a shared workspace where multiple related projects can live under one umbrella. Within an organization, we can create multiple repos, each one dedicated to a specific project. This helps us keep repos organized and separate, so it’s easier to find what you need and collaborate within focused groups.

  • Organizations also let us manage access carefully. We can have private and public repos within a single org. Public repos are open to anyone, which is great for sharing code broadly or collaborating with external partners. Private repos restrict access to invited team members or members from your organization, which is ideal for sensitive internal work.

  • We recently set up the WA-Emerging Infections Program (WA-EIP) organization. The goal is to house all EIP-related work, including both the RESP-NET repositories and the MPOX vaccination study repos.

    • For RESP-NET specifically, our long-term goal is to maintain both private repos for collaboration with internal teams (RHINO, Data Engineering, etc.) and with LHJ partners while also building out public-facing repos where we can share scripts with other RESP-NET sites and the CDC.

  • We’re also building on lessons learned from the Northwest Pathogen Genomics Center of Excellence organization, which Frank stood up. That space is used to collaborate across the molecular epi team, DIQA, and external academic partners.

Now that I’ve shared a high-level overview of all the benefits of git and github and how DIQA uses it, I’m going to hand it over to Frank who will demo how to actually use these tools in practice.

how to use Git/GitHub in practice

• 1. clone
• 2. create a branch
• 3. commit
• 4. push
• 5. pull request
• 6. pull

• find a repo in GitHub
• clone the repo: copy the whole repo to a folder on your laptop

• make your own branch for your isolated edits
• keeps the main branch clean

• write code like normal
• save your code updates, and commit code updates to branch
  • commits are changes stored in the branch
• make small, frequent commits if possible

• after you save and commit,
• push branch to GitHub
  • if first time creating a branch, must publish first
• like taking your local changes and pushing them to GitHub where your team can see the changes

after you push the branch, create a pull request

in GitHub you can pull your feature branch into the main branch - this merges the changes into your codebase

• back in your Git Desktop App, pull the latest version of a branch
• it will pull the latest changes your teammates made into your local clone
• should pull regularly throughout the process

DOH-EPI-Coders

• Our private DOH repos
• Can request for a license
• Teams channels for Q/A and discussions

viewof scan = Inputs.button([
        ["Base Timeline", value => 1],
        ["Top Language", value => 2],
        ["Top 5 Repos", value => 3],
        ["Size", value => 4]
 ],
        {value: 1, label: "Click Plot Type:"}
);

node_data = FileAttachment("repo_activity.csv").csv()


nodes = node_data.map(d => Object.create(d))

// bfScale = d3.scaleLinear()
//   .domain([1, 5])
//   .range([1930, 2020])
//   .clamp(true)
    
// scan = crTriggerIndex

chart_param = ({
  width: width - 800,
  height: 600,
  margin: {
    top: 50,
    right: 40,
    bottom: 80,
    left: 60,
    center: 150
  }
})

chart = {
  // Define base scales for positioning circles
  const x = d3.scaleLinear()
    .domain([0, 1])
    .range([chart_param.margin.left, chart_param.width - chart_param.margin.right]);

  const y = d3.scaleLinear()
    .domain([0, 1])
    .range([chart_param.height - chart_param.margin.bottom, chart_param.margin.top]);

  // Initialize SVG container
  const svg = d3.select(DOM.svg(chart_param.width, chart_param.height));

  // Append title and subtitle
  svg.append("text")
    .attr("x", chart_param.width / 2)
    .attr("y", chart_param.margin.top - 25)
    .attr("text-anchor", "middle")
    .attr("font-size", "20px")
    .attr("font-weight", "bold")
    .text("Beeswarm Plot of GitHub Repos Over Time");

  svg.append("text")
    .attr("x", chart_param.width / 2)
    .attr("y", chart_param.margin.top - 10)
    .attr("text-anchor", "middle")
    .attr("font-size", "14px")
    .attr("font-weight", "normal")
    .text("A visualization of repositories in the DOH-EPI-Coders organization");

  // Preprocess data: Map any language that isn't "R" or "Python" to "Other"
  node_data.forEach(d => {
    if (d.language === "Jupyter Notebook") {
      d.language = "Python";
    } else if (d.language !== "R" && d.language !== "Python") {
      d.language = "Other";
    }
  });

  // Group nodes by language using d3.group
  const languages = d3.group(node_data, d => d.language);

  // Viridis colors for languages
  const colorScale = d3.scaleOrdinal()
  .domain(["R", "Python", "Other"])  // List of languages you want to color
  .range(["#5cc870", "#3B528B","#440154" ]); // Adjusted Viridis colors with more green

  // Scale for node radius based on the number of commits
  const radiusScale = d3.scaleLog()
    .domain([1, 5000])  // Adjust the domain to your data range
    .range([5, 17]);      // Adjust the range for the circle radius

// Scale for node radius based on the number of commits
  const radiusLinear = d3.scaleLinear()
    .domain([1, 5000])  // Adjust the domain to your data range
    .range([1, 90]);      // Adjust the range for the circle radius

  // Define x scale based on create_date for grouping by date
  const xScale = d3.scaleTime()
    .domain([new Date("2020-01-01"), new Date("2026-01-01")]) // Set date range
    .range([chart_param.margin.left, chart_param.width - chart_param.margin.right]);

  // Set up the y-scale based on language groups
  const yScale = d3.scaleBand()
    .domain(["Other", "Python", "R"])  // fixed order  // Use the language groups as domain
    .range([chart_param.margin.top, chart_param.height - chart_param.margin.bottom])
    .padding(0.1);  // Add padding for spacing between the groups

  function createNodes(scan) {
    // Sort repos by commits in descending order and get the top 5 for scan == 3
    const topRepos = scan === 3 ? node_data.sort((a, b) => b.commits - a.commits).slice(0, 5) : [];
    const topRepoCommits = new Set(topRepos.map(d => d.commits));

    // Initialize simulation with the base forces
    const sim = d3
      .forceSimulation(node_data)
      .force("x", d3.forceX(d => xScale(new Date(d.create_date))))  // Position along the X-axis based on create_date
      .force("collide", d3.forceCollide().radius(d => radiusScale(d.commits) + 1).strength(0.5));  // Default collision force

    // If `scan > 1`, apply additional forces for language grouping
    if (scan == 1) {
          // For `scan === 1`, apply the default force with no language division
      sim.force("y", d3.forceY(chart_param.height / 2))  // All nodes at the center of Y-axis
        .force("collide", d3.forceCollide().radius(d => radiusScale(d.commits) + 1).strength(0.5));  // Default collision force

      // Create x-axis for years
      const xAxis = d3.axisBottom(xScale).tickFormat(d3.timeFormat("%Y"));
      const xAxisGroup = svg.append("g")
      .attr("transform", `translate(0, ${chart_param.height - chart_param.margin.bottom})`)
      .call(xAxis);
    
    // Style x-axis labels (make them bold and larger)
    xAxisGroup.selectAll("text")
      .attr("font-size", "16px")    // Set font size to 16px or any value you prefer
      .attr("font-weight", "bold"); // Make the labels bold
    } else if (scan == 2 || scan == 3){
        // Apply additional y-force to divide nodes by language
      sim.force("y", d3.forceY(d => yScale(d.language) + 70))  // Position nodes along y-axis based on language
        .force("collide", d3.forceCollide().radius(d => radiusScale(d.commits) + 1).strength(0.8));  // Adjust collision force

      // Create x-axis for years
      const xAxis = d3.axisBottom(xScale).tickFormat(d3.timeFormat("%Y"));
      const xAxisGroup = svg.append("g")
      .attr("transform", `translate(0, ${chart_param.height - chart_param.margin.bottom})`)
      .call(xAxis);
    
        // Style x-axis labels (make them bold and larger)
        xAxisGroup.selectAll("text")
        .attr("font-size", "16px")    // Set font size to 16px or any value you prefer
        .attr("font-weight", "bold"); // Make the labels bold

        // Create y-axis for language groups
        const yAxis = d3.axisLeft(yScale);
        const yAxisGroup = svg.append("g")
        .attr("transform", `translate(${chart_param.margin.left}, 0)`)
        .call(yAxis);
        
        // Style y-axis labels (make them bold and larger)
        yAxisGroup.selectAll("text")
        .attr("font-size", "15px")    // Set font size to 16px or any value you prefer
        .attr("font-weight", "bold"); // Make the labels bold
    } else if (scan==4){
        // Apply additional y-force to divide nodes by language
      sim.force("y", d3.forceY(d => yScale(d.language) + 70))  // Position nodes along y-axis based on language
        .force("collide", d3.forceCollide().radius(d => radiusLinear(d.commits) + 1).strength(0.2));  // Adjust collision force

      // Create x-axis for years
      const xAxis = d3.axisBottom(xScale).tickFormat(d3.timeFormat("%Y"));
      const xAxisGroup = svg.append("g")
      .attr("transform", `translate(0, ${chart_param.height - chart_param.margin.bottom})`)
      .call(xAxis);
    
        // Style x-axis labels (make them bold and larger)
        xAxisGroup.selectAll("text")
        .attr("font-size", "16px")    // Set font size to 16px or any value you prefer
        .attr("font-weight", "bold"); // Make the labels bold

        // Create y-axis for language groups
        const yAxis = d3.axisLeft(yScale);
        const yAxisGroup = svg.append("g")
        .attr("transform", `translate(${chart_param.margin.left}, 0)`)
        .call(yAxis);
        
        // Style y-axis labels (make them bold and larger)
        yAxisGroup.selectAll("text")
        .attr("font-size", "15px")    // Set font size to 16px or any value you prefer
        .attr("font-weight", "bold"); // Make the labels bold
    }

    // Restart the simulation to apply the changes
    sim.alpha(1)
      .alphaDecay(0.05)
      .restart();

    // Bind data and draw nodes
    const node = svg.selectAll(".node")
      .data(node_data)
      .enter()
      .append("circle")
      .attr("class", "node")
      .attr("r", d => {
            if (scan === 4) {
            return radiusLinear(d.commits);  // Use the linear scale when scan is 4
            } else {
            return radiusScale(d.commits);  // Use the original scale for other scan values
            }
        }) // Set the radius based on the 'commits' field
      .attr("cx", d => xScale(new Date(d.create_date)))  // Set initial x position based on date
      .attr("cy", d => scan > 1 ? yScale(d.language) : chart_param.height / 2)  // Correct y position based on language
      // style('fill') overrides the circle color. use .attr('fill') for conditional colors!
      // .style("fill", (d) => colorScale(d.language))  // Highlight top 5 repos with orange
      //.style("opacity", (d) => topRepoCommits.has(d.commits) ? 1 : 0.6);  // Lower opacity for non-top 5 repos
      .attr("fill", (d) => topRepoCommits.has(d.commits) ? "#FDE725" : colorScale(d.language))
      .attr("stroke", (d) => topRepoCommits.has(d.commits) ? "black" : "none")
      .attr("stroke-width", (d) => topRepoCommits.has(d.commits) ? 2 : 0);

    // Add tooltips with repo info
    node.append("title")
      .text(d =>
        `Repo: ${d.repo}\n` +
        `Commits: ${d.commits}\n` +
        `Contributors: ${d.contributors}\n` +
        `Create Date: ${d.create_date}`
      );

    // Hover effect to change circle color to red on mouseover, revert on mouseout
    node.on("mouseover", function(event, d) {
      d3.select(this)
        .transition().duration(200)
        .attr("fill", "#FDE725")
        .attr("stroke", "black")
        .attr("stroke-width", 3);
    })
    .on("mouseout", function(event, d) {
      d3.select(this)
        .transition().duration(100)
        .attr("fill", (d) => topRepoCommits.has(d.commits) ? "#FDE725" : colorScale(d.language))
        .attr("stroke", (d) => topRepoCommits.has(d.commits) ? "black" : "none")
        .attr("stroke-width", (d) => topRepoCommits.has(d.commits) ? 2 : 0);
    });

    // Show detailed data on click with line breaks
    node.on("click", function(event, d) {
      const clickTooltip = d3.select("body").append("div")
        .attr("class", "click-tooltip")
        .style("position", "absolute")
        .style("visibility", "hidden")
        .style("background", "rgba(0, 0, 0, 0.7)")
        .style("color", "white")
        .style("border-radius", "4px")
        .style("padding", "10px")
        .style("font-size", "14px")
        .html(`
          <strong>Repo:</strong> ${d.repo}<br>
          <strong>Commits:</strong> ${d.commits}<br>
          <strong>Contributors:</strong> ${d.contributors}<br>
          <strong>Create Date:</strong> ${d.create_date}
        `);

      clickTooltip.style("visibility", "visible")
        .style("top", `${event.pageY + 10}px`)
        .style("left", `${event.pageX + 10}px`);

      // Close the click tooltip after 3 seconds (optional)
      setTimeout(() => {
        d3.select(".click-tooltip").remove();
      }, 3000);
    });

    // Update circle positions on each tick of the simulation
    sim.on("tick", () => {
      node
        .attr("cx", d => d.x)
        .attr("cy", d => d.y);
    });
  }

  // Main logic to check `scan` value and call createNodes accordingly
  createNodes(scan);  // Pass `scan` to createNodes to handle the different plot configurations

  return svg.node();
};

Examples

• scripts to create data files (lineage classifications)
• data processing scripts

GitHub Pages

• GitHub can host static webpages for free
• Works well with Quarto and Rmarkdown
• Many of our public repos use GH Pages

Package Documentation

ELR Documentation

Reports

Publications

Resources

• Git tutorial
• GitHub Intro
• GitHub for Beginners Playlist
• DOH GitHub Policies
• Internal Sharepoint Docs (ask for links)
• This slide show’s GitHub Repo

Questions?

Advanced Settings

• Release cycles
• Git Actions