CMSE 495 logo
CMSE495 Data Science Capstone

Code must be Safe, Portable, Reproducible, Robust and Literate

All members of your team must review the Guidelines for Project Code Repositories and you will be graded on how well your team follows these guildlines.

Git Repository Organization

Git Logo

Your git repository should be easy for teammates, instructors, and future maintainers to understand.

Understanding Git

It is assumed that you have been introduced to git in previous classes. We will be using it a lot in this class. Students that are uncomfortable with git may want to review the following tutorials. It is the entire teams responsibility to help your fellow members learn and use git properly:

Another fun resource is the following game which tries to teach you git:

Minimum Repository Structure

ProjectName/
  README.md
  .gitignore
  LICENSE
  src/
  docs/
  data/
  notebooks/
  tests/
  environment.yml or requirements.txt
  INSTALL.md

Adjust names for your project, but keep structure intentional and documented.

Core Expectations

Tooling and Collaboration

Use issues and pull requests for substantial changes. Keep changes reviewable and describe why they were made.

Privacy and Access

If your project has NDA/IP constraints, use an approved private repository configuration and ensure instructors have access.

2. Git Repository

Create a git repository and share it with your teammates and instructors. This repository will be used for the remainder of the course to track all of your code developed for this project. By the end of the semester your repository needs to be well organized and easy to navigate. Use best practices established by other communities. For example, here is a structure you may want to follow:

ProjectName/
    .gitignore
    docs/
         package_name/
              module1.html
              module2.html
         images/
              image1.jpg
    environments.yml
    Examples/
          datafile1.csv
          datafile2.tiff
          datafile3.xls
    LICENSE.txt
    makefile
    package_name/
          __init__.py
          module1.py
          module2.py
          test/
              __init__.py
              test_module1.py
              test_module2.py
    README.md
    setup.py

Projects structure will vary. The important part is that repositories are well documented so that it is easy to navigate.

Again, the README file is extremely important and should be the first “touch point” in your files in your repository. Write the readme in a way that assumes that people reading it know nothing about the project. This means you include a brief description of the class and then guild people to the other parts of the project. All files should be documented and clearly labeled. There should be no confusion about what a file is and why it is in the folder.

Examples

Here is an example projects you can download from a different course with a similar structure. Note these may have had slightly different expectation and assignment goals relative to your project but are still good representation of the types of projects that would be acceptable for this course. Projects shared with permission by former students:

Even more projects (may require logging into MSU GitLab to access:

GitHub vs GitLab vs Other

If your code is under and IP agreement then you are required to use the MSU Gitlab (https://gitlab.msu.edu) server for hosting your private git repository. This requirement can be waived if you have written permission from the IP owner asking that you use another git service (as long as your instructor has access).

If your code is not under an IP agreement then your instructors are flexible as to which online git source you use. Here are things to consider:

Talk to your instructors if you have any problems.

Basic git INSTRUCTIONS

For this milestone we only need to set up the basic structure with the following. To start, please keep the files simple and do not include files that do not add something to the project (see “what not to include” below):

ProjectName/
    .gitignore
    LICENSE.md
    README.md

Here is a description of each of these:

HINT: Many of you may find this git template helpful.

If you need help figuring out how to set up your git repository there are a ton of tutorials online. For example here is a good one:

If you continue to need help go see your instructors.

The following video are instructions specifically for how to use the MSU Gitlab. We will be using the MSU gitlab for all projects because it allows us to best maintain file permissions. If you have a completely opensource project with no NDA or IP agreement you are also allowed to post on Github or other public spaces:

Some of you may get some sort of “Authentication” error when trying to use git on a windows machine (especially if you have your computer already set up to use github). If that is the case, the following video may help you set up an SSH key on your windows machine.

As you update and change the files in your repository you will need to push those changes to gitlab. The following instructions walk you though this process:

What not to include (building a .gitignore file)

First thing we want to remember is that not everything should go into a git repository. i.e. we do not want to bloat our repository with unwanted files. The git repository works best with Text files that represent “source” code and not compiled or generated code. Here are some basic guidelines of what not to include:

A good rule of thumb is that if you did not generate the file and/or do not know what it is you probably do NOT want to include it in your repository.

WARNING do not blindly add all files to your repository with the * (star) syntax. This is bad practice. For example do NOT do the following:

git add *

Other files to avoid

In addition to the above files it is good to avoid any type of “Binary” file (with a few exceptions). As stated early, git works best with text files so it can easily track changes. Some example binary files to avoid include:

Note: one exception to the above rules are image files (ex jpg or png) that are used to help markdown or in the documentation. It is typically okay to include these since they tend to get included only once and do not change much as the project evolves.

.gitignore file

The .gitignore (read “dot git ignore”) file is used to help keep unwanted files out of your project. Each line .gitignore file are filenames you want git to ignore. For example, based on what we said above, a good place to start on your .gitignore file would be the following two lines:

.ipynbcheckpoint
__pychache__

What should go into a .gitignore depends a lot on the type of project. However, you don’t need to invent these from scratch. For example, you could just copy the .gitignore file from the course repository or find one on the internet.

README.md file

Make sure your git repository has a README.md file. This file is very important as it will be the first place everyone will look when they first get your repository. Write this as if someone in the future stumbles across your folder. Include a brief description of the class, the community partner, the project and the goals of this repository. Link to any installation instructions or getting started instructions. It is important that this file guides readers to the rest of the repository. A great resource for writing a well thought out README file can be found here:

https://readme.so/editor

Jupyter notebook files in git repositories

Turns out that Jupyter notebook files and git repositories work very poorly together. Jupyter notebook files are a unique combination of source and program generated information. So, every time you run a jupyter file it can add output cells which make git think you you changed something important. In many cases it is just a few numbers or some output text. When you run the git status command it always looks like jupyter notebook files have changed even when they have not changed.

A good rule of thumb is to clear all of the output files before committing any changes to jupyter notebook files.

The following video goes though why we have to treat jupyter notebooks this way:

Direct Link

Licensing file

As authors of software it is important to let people know how they can use our software.

If your project has an IP agreement you should include an unsigned copy of the agreement and include a LICENSE file similar to the following.

License Agreement

This software is the intellectual property of [COMMUNITY PARTNER NAME] and was developed by the [TEAM NAME] Capstone team, consisting of [Name1], [Name2], [Name3]...

The use of this software is governed by the terms and conditions outlined in the attached Intellectual Property Agreement (IP Agreement) identified as [FILENAME].

By using this software, you agree to comply with the terms of the IP Agreement.

If you do not have an IP agreement that covers the License, the following article is a great resource for learning the types of terminology and logic used when talking about software License.

Creative commons license

Include a ./LICENSE test file in your top directory. Select which license to use using the following website:

Copy and paste your chosen license file into a file named ./LICENSE

The following articles are a great resource for learning the types of terminology and logic used when talking about software License.

Make sure you talk to your instructors to know what type of license is appropriate for your project given. What license you use may also depend on the IP agreement made with you and the community partner.

Include a ./LICENSE.txt test file in your top directory.

3. Working with files professionally

Anytime you are working with files you need to pay attention to what you are doing. Pay attention to whom you will be sharing files and keep things clean and professional. Points will be taken off for unprofessional files.

Consider the following guidelines.

Use Filenames that make sense

Make sure the contents of the files are clear. If something is associated with a specific date then would like to use the following format:

YYYYMMDD-TEAM-Contents.ext

DO NOT use spaces in file names

I said this above and I will say it again.

When you name all of the files and folders inside of a repository, it is important that your names DO NOT include spaces. Although all modern computer’s have ways to accept names with spaces do not use them. Instead use underscores (_) or CamelCase (No spaces and capital letters at the beginning of each word in the name). Avoiding spaces in your names will Always save time in the long run.

Always Use Relative Paths

In your code there are two basic ways to determine the location of a folder inside your computer; Relative Paths and Absolute Paths. A relative path is a path starting from your current directory and an absolute path is is a path starting from your computer’s “root” directory.

ALWAYS use relative paths in your git repository. This ensures that others will be able to use your software if they download it onto their computer. For example:

Good: ./data/  or ../data/ is a relative path to a child directory or sibling directory called data. 

mBad (not acceptable): C:/research/data or /mnt/home/data are absolute paths to a data directory