CSE-506 Lab Facilities (Spring 2019)

Introduction

We have set up limited lab facilities for use by registered class students during the semester. The lab's machines are accessible only remotely and via a secure shell. Details are provided below. Note that the procedures we describe here will be updated and adjusted as needed throughout the semester.

There are restrictions on the use of the Lab facilities:

The lab facilities are limited. Start your homework early! Don't wait until the last minute. The more people compile code all at once, the slower the lab facilities will be. Vary your time of work: some people prefer day-time work, some like to work late at night, some over the weekends.

Getting Initial Access

To be able to do your homeworks, you need to do a few things. They can be done independently.
  1. First, you'll need to ensure that you have physical access and login access to somewhere on campus, where you can sit down and do you work. You can use the Grad CS Labs, SINC sites, a personal workstation or laptop, etc.
  2. Second, you must subscribe to class mailing list. A lot of useful information will be provided via the mailing list. Through the list, we can distribute information much more quickly to the class students than during the weekly lectures.

    Note: I strongly encourage you to subscribe with your Stony Brook CS account. This will ensure that mail will arrive to you more quickly, and that you are less likely to run out of your mail quota than if you were using some other ISP's mail service. If I post an important message to the class list, and you don't get it because your mail quota has exceeded, I may get a bounce back saying so, but I will have no way to communicate back with you. It is therefore your responsibility to ensure that you can always read the class mailing list and to stay current with what I post there.

    Finally, please do not set your list subscription mode to Digest because digests can take several days to get emailed to you.

  3. Third, you need an account on the OSLAB network. If you are a registered student in this class, your account is already active. You don't need to apply specially for it.

  4. Fourth, you need remote login access to the OSLAB lab that includes the machines on which you will program and test your programs. Note, you will not get physical access to this lab. You will need two forms of access: SSH to your personal virtual machine, and a Remote Desktop (rdesktop) method to the VMware ESX Console. Both are described below in greater detail

    Please note that you will get an OSLAB account that will allow you to log into several machines. Do not send email to the TA or the instructor from these machines, nor should you expect to get email sent to you on these machines; they are not configured for email processing. Please do all your email communication through whatever email means and email ID you were using outside these labs (e.g., CS's MS Exchange email system). It is particularly important that you follow these instructions to ensure that you could communicate with the teaching staff promptly.

Remote Desktop (RDP) Access to your Virtual Machines

Each of you will get your own personal virtual machine that you'd be able to reboot and shutdown remotely. You'd be able to SSH into your VM, download kernel sources, compile them, install new kernels, etc. However, when developing kernel code, you sometimes need console access to select from multiple installed kernel versions, reboot into single-user mode, run file-system checker (fsck), and more. To get console access to your VM, based on your own OS (Windows, OS X, Linux), please follow the instructions here.

Note, find out which ESX server you were assigned to. So you can login to that server for your VM. Consult this assignment list.

Using Your Personal VM

Once you managed to login to the VMware vSphere GUI and start your VM, check the "Console" tab to see the progress how your VM is rebooting. The default VM is a small Linux system setup for you. The Linux VM was setup to pick a random IP address via DHCP. Once booted, the system will prompt you to login, something like: vl321 Login: The above means that your VM got the dedicated hostname called "vl321.cs.stonybrook.edu". Every student VM will get a different "vlXXX" dynamically assigned DHCP address; note the actual hostname that your VM gets and be sure to login only to that one. Note that the host name starts with a lower-case "V", then a lower-case "L" (not the digit one - "1"). You can ssh directly to it as follows: ssh -p 130 root@vl321.cs.stonybrook.edu Note: the default ssh server runs in port 130, to reduce the number of casual attacks that our department sees every day by so-called "script kiddies". No matter what SSH client you use, you have to ssh using port 130, not port 22 (you will get a "connection refused" error if you try to connect with the default SSH port of 22).

Note also that you're logging in directly the unix root superuser! You have to be very careful when using the root user, as it is easy to damage your VM. For that reason, the VMs you get by default already have a base snapshot, and you're encouraged to take more snapshots of your VM as needed. Keeping the last couple of snapshots around is useful, but please also don't start snapshotting every five minutes.

You can only login to the vlXXX machine using root, not your AD password. The default root password for your VM will be given to you in class. Please do not share it with anyone else. And please change it as soon as you login, to prevent others from logging into your VM inadvertently.

WARNING: Do not share the root VMware password with anyone else outside class!!! In addition, do not use your vmware root privileges for anything other than to perform your homework assignments for this class. Any deviation or abuse of your privileges will be result in your account disabled, a grade of "F" for the class, and a referral to the Academic Judiciary Committee.
Lastly, please be nice to your classmates. When you are not actively working on your VM, please shut it down or suspend it; that will release more memory and CPU resources for other VMs. You can bring always bring you VM back up quickly at any time later.

Troubleshooting when your VM no longer starts

The infrastructure we have for you will ensure that your VM continues to run even if your ssh connection to it was severed. All you have to do is reconnect to the vSphere console and view your VM's console. During the course of the semester, however, you're likely to install buggy software, crash your kernel multiple times, etc. This could render your VM's file system so corrupt that it won't boot any more. To help you recover from this, the VMs you get by default already have a base snapshot: in case of trouble, you can revert to it. It is advisable that each time you've managed to get a major milestone, that you take another snapshot and label it properly to explain what it has. This'll allow you to go back and forth to any state of your VM, and possibly retrace your steps, or redo something over. So take as many snapshots as you'd like, but there's no need to take a snapshot every five minutes: that'll consume too much disk space and eventually you will exceed your quota. It is customary to keep a small number of key snapshots, and to delete others which are no longer deemed necessary.

Using your own personal Linux machines

We strongly recommend that you do all of your homework assignments on the OSLAB VMware machines. That way you can benefit from our experience using these machines, as well as the experiences of other students in the class.

Nevertheless, a number of people experienced with Linux sometimes choose to work on their assignments elsewhere: at home, at work, in a dorm, on a personal laptop, etc. That is OK. However, it is solely your responsibility to ensure that your code is working and properly committed on the OSLAB VMware machines! If you choose to develop your code outside the OSLAB, be sure to give yourself enough time to migrate it into the VMware machines, test it there, debug it as needed, commit it to CVS, check out a test copy and test it, etc. Don't assume that just because it worked for you outside the lab, that it'll automatically work in the lab: there could be many reasons why your code may not work in other settings. We will not allow people to bring laptops or other machines to demo their code. We will not accept excuses such as "but it worked for me at home" or "it really works --- I just need to copy it over."

If you do choose to work elsewhere on your assignments, we recommend that you duplicate the same exact environment we have on the VMware machines (Red Hat Linux 9 with all of the latest patches, configured properly). We will try to help you to set up your personal machine appropriately, but please realize that our time and resources are limited to official class matters. Unfortunately, we do not have the time to help each student in the class to set up a separate, personal work environment, especially on esoteric machines. That's why we worked hard to prepare a standard work environment using VMware for the entire class.

Compiling Kernels and Other Sources

The OSLAB personal VMs we give you only provide root login. You can create other accounts if you'd like. But most of the work kernel developers do is don as root, so you should get comfortable working with (dangerous) administrative privileges.

The following annotated example shows you step by step the commands needed to build a Linux kernel:

  1. SSH to your VM:
    ssh -p 130 root@myhost.cs.stonybrook.edu
    
    where myhost is the DHCP hostname of your VM (e.g., "vl321" as shown above). You can only login as root to the dNNN machine, with the password that will be given in class; don't try to use your AD password.

    Note: from now on, all your commands should be executed inside your own virtual machine, the one you just rebooted.

    cd /usr/src
    
  2. Retrieve the Linux kernel sources via GIT. Each of you have been given a personal GIT repository for each homework assignment.
    git clone ssh://user@scm.cs.stonybrook.edu:130/scm/cse506git-s19/hw1-user
    
    where user (twice in the above command) is your AD username. You'll be prompted for your CS AD password.

    The initial checking out (or "cloning") of the git repository will take some time. Using your checked out repository afterward will be faster. Note that the first assignment must be done in Linux version 4.x (where "y" is whatever's the latest sub-release). The source code repository cloned for each student already uses a specific kernel version for you to use. Subsequent assignments may be required to use a different version of the kernel. In that case, replace "4.x" in the instructions the follow with the newer version of the kernel.

  3. Change your working directory into kernel source directory
    cd hw1-user
    
  4. Configure a kernel for running on VMware
    CAUTION: While I provide some detailed instructions below, your general goal is to build the smallest kernel you can, with as few features as possible, such that it does not failed to boot or produce any errors during the boot of the (CentOS) Linux image. If you take off too many features, your system may not boot, or will boot with errors. Note that sometimes it is also important whether a feature is enabled as a loadable module or compiled statically into the kernel image.
    Use the command below to configure your kernel. You can also use "make config" for line-by-line configuration.

    make menuconfig
    

    Note: if you find that certain packages are missing on your VM, you can install them using the "yum" utility. For example, to fully run the menuconfig option, you may have to install the Ncurses development library with:

    yum install ncurses-devel
    

  5. To build the kernel and modules (also can take time), run
    make
    
At this point, your kernel and modules are built, and you're ready to install them.

Note: in the following instructions, commands which appear in red are executed as root. Then run these commands:

  1. Become root using the root password given to you in class
    /bin/su -
    
  2. Install modules:
    
    make modules_install
    
    
  3. Install the main kernel image:
    
    make install
    
    
  4. Normally, "make install" will update your kernel boot file which lists all of the kernels you can boot. If (and only if) it doesn't, then you'll have to update it yourself. Using a text editor such as emacs (or vi, pine, jed), add a kernel boot entry to the end of the boot loader configuration file grub.conf:
    
    emacs /etc/grub.conf # or grub2.conf
    
    
    The entry you can add has the following text:
    
    # to boot from the first IDE drive, first partition
    title test kernel (2.6.30.5 ide)
    	root (hd0,0)
    	kernel /boot/vmlinuz-2.6.30.5 ro root=/dev/hda1
    # to boot from the first SCSI drive, first partition
    title test kernel (2.6.30.5 scsi)
    	root (hd0,0)
    	kernel /boot/vmlinuz-2.6.30.5 ro root=/dev/sda1
    
    
    Be careful not to change anything else in grub.conf, or you could render your system unbootable and then you'll need to reinitialize your vmware. In particular, leave behind the original entry for the default Red Hat kernel. When you're done adding the new kernel entry to grub.conf, save the file, and exit the editor.
  5. Finally, reboot your system
    
    reboot
    
    
    When the system will boot, you will see a GRUB menu that now lists two possible kernels to boot. The second (default) one is the one that came with the system and is always good to boot into as a backup solution. The first entry should be the one that you just installed. Use the arrow keys to select the first one, the one you just installed, (within 10 seconds) then hit ENTER to select it. Your system should now boot your new kernel. When it comes up, login as yourself to ensure that everything appears to be in working order.
All of the procedures described in this section are just to get a new, generic, unmodified (i.e., vanilla) kernel built, installed, and booted. Now that you've done that, you can move on to trying to build sources for the homework assignment, either as independent modules or as modified kernel sources, and try your changes.

GIT: Submitting, committing, and backing up your code

This section includes information on how to submit your code via GIT. This is not a comprehensive guide to GIT, so consult the full GIT manual for more help. GIT is a superior Source Control Management (SCM) system than both CVS and Subversion.

Do not wait until the last minute before submission time to figure out how to submit your homework assignment. Learn these procedures well ahead of time, experiment with them, and try and try again. You can submit your assignment as many times as you'd like, but we will consider the last set of files you've submitted as the ones you want us to grade. Don't be late because we won't accept late submission excuses such as "GIT didn't work for me."

Warning: protect your homework from accidental loss! All of the files you have in your VM directories are not being backed-up. That means that if you remove your files my mistake (happens a lot, often right before a due date), you will lose all of your data and won't have anything to submit. Using GIT you can make a copy of your files and more: record each version of your files as you change them. It is your responsibility to use GIT often. The graders will not accept excuses such as "I just removed all my files and can't submit my homework."
To simplify matters for you we've set up GIT repositories for everyone.
  1. SSH to your VM:
    ssh -p 130 root@myhost.cs.stonybrook.edu
    
    where myhost is the DHCP hostname of your VM (e.g., "vl321" as shown above).

    cd /usr/src
    
  2. Retrieve the Linux kernel sources via GIT. Each of you have been given a personal GIT repository for each homework assignment.
    git clone ssh://user@scm.cs.stonybrook.edu:130/scm/cse506git-s19/hw1-user
    
    where user (twice in the above command) is your AD username. You'll be prompted for your CS AD password.

    The initial checking out (or "cloning") of the git repository will take some time. Using your checked out repository afterward will be faster. Note again that this checkout procedure must be done only once: repeating it can confuse GIT into using recursive repositories, and your homework could be lost.

  3. Next, you can modify any existing source file, then type "make" to compile your changes, install them, and test them.

  4. If you add new files to the repository, you will have to tell GIT to commit them in the repository:
    git add newfile.c
    
    If you have other files to include in your submission, you should "git add" them as well. But, do not submit or add/commit binaries or other object files: those can be produced easily by running "make".

  5. If you wish to create subdirectories under a certain GIT repository, you can do so as follows. For example, to add a subdirectory "test":
    mkdir test
    git add test
    
  6. You can check your uncommitted changes using:
    git diff
    
  7. You can list which files have changed using:
    git status
    
  8. When you are ready to commit changes, you first have to update the GIT index of any changed file:
    git update-index changedfile.c
    
  9. After updating the index of the file, you will have to commit it to your local repository:
    git commit
    
    Be sure to provide a suitably detailed commit message to explain what you did.

  10. Note: at this point, all git did is commit your changes to the local repository only. These changes are NOT safe from VM corruption (and if you revert to an earlier snapshot, you will lose all your committed changes!). To ensure that your changes have made it to the remote (and save) copy on the "scm.cs.stonybrook.edu" machine, you have to "push" the changes:
    git push
    
  11. You can also check what was committed using
    git log | more
    
There are many more GIT commands you could use. Check the documentation online, man pages, and more.

Lastly, you can check out a second, temporary version of your own committed files and verify that they compile and run as you expect them to:

cd /tmp
git clone ...
cd hw1-user
git log | more
# configure/test kernel, etc.

Last Updated: 2019-02-26