TweetFollow Us on Twitter

December 96 - Be Our Guest:

Be Our Guest:
Source Code Control for the Rest of Us

D. John Anderson and Alan B. Harper

Imagine you're working on a program than runs on both Macintosh and Windows computers. Suppose you aren't the only programmer working on the project, and you occasionally want to work from home. Seems like a pretty common situation.

Soon everyone begins to make their own changes to the program and you realize that you need some kind of source code control. When people think of source code control, they often picture a safe database that keeps a history of every change to the project. Most source control systems do a reasonable job of that. But what surprisingly few people worry about is the much greater need to merge each programmer's changes into the common code base. For some reason this is where many popular source code control systems fall flat; here we'll tell you about one that doesn't.

Many systems require that you check out a file before modifying it. This is so that two people don't modify the same file at the same time. Unfortunately, this makes global changes difficult. Something as simple as changing a routine name from doQMFX to PrintFile doesn't get done because you need to check out every file and someone else already has them checked out; it's just not worth the hassle. In the early stages of development, when everyone is changing many files at the same time, the check-in/check-out model can slow you down even more. Some systems let a file be checked out by several people, who must then merge their changes into a single file. Even with the best tools, this process of merging often requires you to look at every individual change, presumably so that you can verify each one. But you typically end up incorporating every change anyway. Why not save a lot of time and headaches by automating this process completely?

There are other annoying problems with some source code control systems, such as limited cross-platform support and difficult or impossible access to the database from home. And with some systems that use a database, you can't do simple things like grep through your files. These problems can really get in the way of getting your work done.

If after reading this far you say, "Yeah, I hate source code control. I wish I just had a bunch of files and didn't have to think about other people stepping on my files," then read on. If instead you say, "I don't get what he's talking about. I don't mind checking files out and back in and merging them by hand," then skip to the next article.


OUR SOLUTION

Still with us? Great. To avoid these problems, we at Eclectus have developed three tools based on the GNU diff utility: Merge, Difference, and Undifference. They accompany this column on this issue's CD and develop's Web site. Here's an overview of how our scheme works:
  1. We put all the files in the project in one directory tree -- named, say, HotApp-0. We mark all the files read-only and put them on a shared server.

  2. Each programmer makes a copy of HotApp-0 on his or her local machine. They do all their work in their own local copy, changing any file they want.

  3. When everyone agrees that their changes are ready to merge into a new version, we use Merge, which automatically incorporates the changes into a new source tree that we name HotApp-1. When two people modify the same line, Merge reports a conflict that we must look at and resolve. Surprisingly, conflicts are rare. After fixing the conflicts by hand, we make any changes necessary to compile the application; then we mark each file read-only and put HotApp-1 on the shared server.

  4. All programmers copy HotApp-1 to their local machines and development continues.

Suppose I want to bring some work home with me. I use Difference to create a difference script of changes between the current version -- say, HotApp-1 -- and my local working copy. This difference script is usually small enough to put on a floppy disk or beam over a modem. At home, where I also have a copy of HotApp-1, I use Undifference to restore the working copy of my project. I can copy my changes made at home back to work the same way.

If I want a copy of the source for safekeeping, I either copy HotApp-1 from the shared server or, if I really care about space, I difference HotApp-1 against HotApp-0 and compress the result, which is tiny.

If the machine I'm working on doesn't get backed up automatically every day, I can use the same method to back up my work. I just difference my working copy against the most recent shared copy, compress it, and put it in a safe place -- like on a server that's backed up automatically. Compressed daily difference files are so small that I can keep a year's worth in only about 20 MB.

Because these tools are based on GNU diff, they're free (thanks to the Free Software Foundation), the source code is available, and they're extraordinarily fast. Using Difference, Undifference, and Merge is approximately as fast as copying an equivalent number of files. We can do a complete n-way merge -- a merge between n programmers -- in less time than it used to take to check out just a few files with competitive alternatives. We can difference or undifference our project (about 4 MB of source) in less than one minute on a low-end Power Macintosh or a Windows NT computer, and merge two trees in less than two minutes.

Because the source is freely available, you can easily tweak the tools to add your favorite features. Of course you'll have to share those features with everyone else, but that just makes the tools better.


A REAL-LIFE EXAMPLE

At Eclectus, our current project is a program that runs on both the Mac OS and Windows platforms, each developed at a different location. Let's say Alan and John are each modifying their own copy of HotApp-0.
  1. When they decide it's time to merge their changes, Alan beams John his diffs against HotApp-0 via e-mail.

  2. John uses Undifference on Alan's diffs to reconstruct Alan's source code tree. Now both John's source and Alan's source are on John's machine.

  3. John merges Alan's source code with his own to create a new version, naming it HotApp-1, and makes any changes necessary to compile HotApp-1 on Windows.

  4. John uses Difference on HotApp-1 against HotApp-0 and beams these diffs to Alan via e-mail.

  5. Alan undifferences these diffs against HotApp-0 to construct HotApp-1, the same new version John now has. Alan might have to make some changes to get HotApp-1 to compile on the Macintosh because of John's recent changes that were automatically merged -- for example, John might have forgotten to ifdef a Windows-specific piece of code. These changes will show up after the next merge of HotApp.
Development continues, and we repeat this process whenever we feel it's time to merge again. We usually use a modem or e-mail to exchange the diffs, since they're small.

This process generalizes to more than two programmers since the merge utility will handle any number of modified versions. Deciding when to merge is up to all the programmers. We usually merge when somebody wants to make their changes available to everyone else and nobody has totally broken their version. Sometimes months pass between merges, sometimes we merge twice a day. Near the end of a development cycle we like to merge at least once a day so that a recent copy is always available for testing.


DETAILS

Automatic merging, tiny diffs files, no time-consuming file checkout? Sounds great, but there are a lot of details you're probably wondering about.

What do the merges look like?

To merge changes, we use a command-line environment (MPW, ToolServer, and so on). In the above example, we would use the following Macintosh command to merge changes:
merge :HotApp-0 :HotApp-John :HotApp-Alan :HotApp-1
In this example, HotApp-0 is a directory tree containing the original code that both John and Alan started with. HotApp-John is a directory tree containing John's version after his changes, and HotApp-Alan contains Alan's version. HotApp-1 will contain the result of the merge when the command is finished. If we had more programmers, we'd just include their directories after Alan's. (With several programmers, you might think that the time involved in sending complete source trees back and forth would be prohibitive, but remember that the only thing that needs to be sent is a small diffs file, and then the source tree can be rebuilt quickly and locally.)

When conflicts occur in the merge, the conflicting files are renamed by prefixing an exclamation point (!) to the filename, and the conflicting lines of code are marked in the merged file. Usually conflicts are solved by editing the merged file and taking one person's changes. After conflicts are resolved, you rename the file back to its original name.

Are automatic merges really safe?

Many programmers are suspicious of automatic merges. What if Joe makes a change that's incompatible with Helen's, but they don't change the same line? Merge doesn't identify this snafu. To avoid or identify bugs caused by incompatible changes, Joe and Helen must talk to each other or look at all their changes. If you're so inclined, you can give Merge an option that will list all the changes everyone has made so that you can review them individually. Even so, after seeing automatic merges being used for over 10 years in lots of different projects with lots of programmers, we've found that the time saved and the elimination of human error from manual merges more than compensates for rare cases of incompatible changes. Of course, as you near the shipping date, it makes sense to have every change to the project carefully code-reviewed.

Can I rename a file?

Yes, as long as you do it right after a merge, when there's only a single copy of the source tree; otherwise Merge treats renamed files as new and they won't get properly merged with the previous source. But since Merge identifies files that were added or deleted from each person's source tree, it's easy to detect these situations.

What happens to junk files?

Imagine you created some temporary file named Junk. Merge lists this file as a newly created file found in your source tree. If you like, these tools can also ignore certain files based on their filename extension. We like to put all our derived files, like object files, in a subdirectory named Derived.i and have Merge, Difference, and Undifference ignore files and directories that end in ".i". Just edit a table in the source code to define which extensions you want to ignore.

What happens if I rearrange a lot of code?

If two or more people modify the same routine and one of them moves the routine to a new place, Merge won't be able to merge the changes automatically. This case is treated the same as if two or more people modified the same line: you have to look at the changes and sort them out by hand. We often delay rearranging a lot of code until after a merge, but before we distribute the merged version.

How do you pick up someone's bug fix?

If someone has a fix to a bug that they're not ready to merge, but it's holding up your work, you can just copy their code containing the fix and put it in your project. Merge doesn't treat this as a conflict even though more than one person changed the code, because their changes were identical.

What about binary files?

Merge automatically handles one person changing a binary file, but not more than one. In the latter case you must merge by hand. For this reason, we try to keep as much of our project as possible in text rather than binary files.

Does this really work with more than two programmers?

T/Maker successfully used our tools for a number of years while developing WriteNow For Macintosh with five programmers. Various companies with many programmers working on multiple projects have been using successive automatic two-way merges for years.

What about cross-platform support?

Merge, Difference, and Undifference are written in C and currently run on the Mac OS (MPW with CodeWarrior 10 tools and libraries), Windows NT or 95 (Microsoft Visual C++ 4.0), and NeXT (GNU C). If you need to support a new platform, these tools should be easy to port as long as your platform has an ANSI C compiler. Some special issues arise when working on cross-platform projects. Let's suppose I have all the code for a Macintosh project and I want to move it to Windows NT or 95.
  1. I difference my source code tree against an empty directory on the Macintosh, resulting in a text file that I move to Windows.

  2. Windows uses different end-of-line characters, so I run a little utility, MacToWin, to change the end-of-line characters. Going in the opposite direction, CodeWarrior IDE and BBEdit on Macintosh can automatically change end-of-line characters.

  3. I undifference against an empty directory on Windows and my files are reconstructed. Even binary files are transferred correctly.

  4. Finally, I name the directory containing the code HotApp-0 on both Windows and Macintosh.
We continue to develop as before, except we each have a separate copy of HotApp-0 on our respective platforms, rather than a shared copy on a single server. If I were to share a single server from both platforms, I would have two separate directory trees, one for each platform, each with the correct end-of-line characters. This would let you use tools like grep in each platform to search through the files in the project. When you need to synchronize the two source trees, you'll convert one of them to the other platform for the merge.

If you work on two different platforms at the same time, you could make a separate copy on each platform and treat each one as if it were owned by a different programmer.

Better yet, if you use CodeWarrior or BBEdit for editing on the Macintosh, the end-of-line problem goes away completely and you can keep just one source tree for both platforms.

What do you do with resource forks on a Macintosh?

When you use Undifference and Merge on a Macintosh, they take the resource fork for the new file from the original directory tree -- that is, HotApp-0 in the above example. This means that after a merge, any changes you made to the resource fork since the last merge must be made by hand. However, once it's in a directory tree that everyone copies -- HotApp-1 in the example -- the resource fork will automatically be propagated to new versions by Undifference and Merge. When you use Undifference and Merge on other platforms to incorporate changes from a Macintosh, resource forks are ignored. This means that you need to store any resources that you regularly edit in a text file and use a resource compiler like Rez.

What about changes to date and time?

Undifference preserves the date and time of a file in a new modified directory tree when there were no changes to the file. This means that you can iterate through many merge cycles and files that weren't changed will still have the same dates and times.

What about non-ASCII characters?

The Mac OS, Windows, and UNIXreg. platforms differ in their interpretation of characters outside the 7-bit ASCII range. These characters are usually not a problem when they're in a file used on only one platform. However, when one of these characters occurs in a file used on more than one platform, it's often a bug. Difference and Undifference identify files that contain non-ASCII characters to help detect this potential problem.

What about long filenames?

When developing on more than one platform, you're limited to the lowest common denominator filename length and path length, since the same file must have the same name on all platforms.

What about change comments?

The tools don't require you to add change comments. If you want them, you must manually add them when you edit your code.

What is the Free Software Foundation?

These tools exist because of the Free Software Foundation. They provide the source code to many useful programs, including the GNU utilities upon which these tools were based. (GNU is short for "GNU's Not UNIX.") Note that any modifications to their code must be made freely available under the same terms. You can contact them via e-mail at gnu@prep.ai.mit.edu.


BEGGING TO DIFFER

These tools grew out of our frustration with existing source code control systems. After using them for many years, we've found them to be indispensable to our development. They've served us well because they're simple, fast, and easy to adapt to new situations. We hope that you'll enjoy using them as much as we do -- and if you don't, perhaps you'll improve them.

D. JOHN ANDERSON (jander@c2.org) and Alan are Eclectus Software. Together they write cross-platform applications for Windows and the Mac OS. Someday they plan to rule the consumer applications market. John lives in La Honda, California, where he writes software outdoors in a large tent with its very own ISDN line. At other times you might see him running or bicycling through the redwoods. His latest hobby, casting molds from faces, was inspired by the video "Better One-Piece Head Molds From Life."*

ALAN B. HARPER (aharper@dnai.com) learned recursive descent from John 15 years ago. His latest accomplishment is a fast cross-platform persistent object store -- which means he can now write programs without worrying about serialization, undo, byte order, garbage collection, or running out of memory. In his off hours, Alan can be seen with other volunteers of the Golden Gate Raptor Observatory following radio-tagged hawks as they migrate through California.*

Thanks to Helen Casabona, Pete Gontier, Andy Jeffrey, and Tim Maroney for reviewing this column.*

develop welcomes guest columns on a variety of subjects. Please submit your column draft or idea to develop@apple.com.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

iShowU Instant 1.3.0 - Full-featured scr...
iShowU Instant gives you real-time screen recording like you've never seen before! It is the fastest, most feature-filled real-time screen capture tool from shinywhitebox yet. All of the features you... Read more
Capo 3.8 - Slow down and learn to play y...
Capo lets you slow down your favorite songs so you can hear the notes and learn how they are played. With Capo, you can quickly tab out your songs atop a highly-detailed OpenCL-powered spectrogram... Read more
Viber 11.7.0 - Send messages and make fr...
Viber lets you send free messages and make free calls to other Viber users, on any device and network, in any country! Viber syncs your contacts, messages and call history with your mobile device, so... Read more
Drive Genius 5.3.1 - $79.00
Drive Genius features a comprehensive Malware Scan. Automate your malware protection. Protect your investment from any threat. The Malware Scan is part of the automated DrivePulse utility. DrivePulse... Read more
CorelDRAW 21.2.0.708 - Graphic design so...
CorelDRAW - professional graphic design software for vector illustration, layout, and so much more. Get started quickly and easily with a wealth of intuitive tools, built-in learning materials,... Read more
Navicat Premium Essentials 12.1.27 - Pro...
Navicat Premium Essentials is a compact version of Navicat which provides basic and necessary features you will need to perform simple administration on a database. It supports the latest features... Read more
OmniFocus 3.4.3 - GTD task manager with...
OmniFocus is an organizer app. It uses projects to organize tasks naturally, and then add tags to organize across projects. Easily enter tasks when you’re on the go, and process them when you have... Read more
Microsoft OneNote 16.30 - Free digital n...
OneNote is your very own digital notebook. With OneNote, you can capture that flash of genius, that moment of inspiration, or that list of errands that's too important to forget. Whether you're at... Read more
Apple Configurator 2.11.1 - Configure an...
Apple Configurator makes it easy to deploy iPad, iPhone, iPod touch, and Apple TV devices in your school or business. Use Apple Configurator to quickly configure large numbers of devices connected to... Read more
Luminar 3.1.4 - Powerful, adaptive, conf...
Luminar is the new full-featured image editor that adapts to the way you edit photos. Over 300 essential tools to fix, edit, and enhance your photos with comfort. The future of photo editing is here... Read more

Latest Forum Discussions

See All

Breakout: Dark Prison is a fast-paced ac...
Breakout: Dark Prison is an action RPG from LaterSoft. Set in the aftermath of a deadly virus outbreak your daughter has been taken from you because she has an immunity to the illness in her DNA. Not being a fan of experimentation on children –... | Read more »
Apple Arcade in review
This weekend, Apple Arcade will officially be one month old. That means anyone who signed up for the free trial on day one has a decision to make: Stick with the service and shell out $5 a month, or cancel and go about your merry way. | Read more »
Alluris is a choose-your-own adventure g...
Alluris is an RPG that the developer's are calling a swipe-your-own adventure game. This is because the game incorporates a Reigns-style - swiping left or right - selection mechanic to make all the decisions you'd usually expect to make across... | Read more »
Hello Hero All Stars receives update wit...
The first Hello Hero game hit global platforms in 2013 and proved a huge success, with developer Fincon adding two more entries to this popular series of casual RPG games since. Released in June this year, Hello Hero All Stars brings many of the... | Read more »
Zombieland: Double Tapper, a cartoon idl...
Zombieland: Double Tapper is the idle RPG tie-in to the upcoming Zombieland: Double Tap. Oddly, it's one of two different Zombieland games launching today, with the other being the Switch title Zombieland: Double Tap - Road Trip. [Read more] | Read more »
Rusty Lake's The White Door launche...
Rusty Lake and Second Maze's intriguing point-and-click adventure game, The White Door, is now up for pre-order on the App Store. This one sees you playing as Robert Hill, a mental health patient who is suffering from severe memory loss. The game... | Read more »
Hellrule is an auto-runner inspired by G...
Hellrule is an upcoming auto-runner game from independent developer Pedrocorp where players will take control of a dapperly dressed gentlemen who comes equipped with a razor-sharp umbrella for slicing up his foes. The game will be available for... | Read more »
Grobo is a gravity bending puzzle platfo...
Grobo is a 2D puzzle platformer that marks the first release from developers Hot Chocolate Games. You'll find yourself manipulating gravity as you make your through this title that's available now for iOS and Android. [Read more] | Read more »
Adrenaline, Compulsive Entertainment’s h...
Compulsive Entertainment’s high-octane arcade racer, Adrenaline, has now made its way to the App Store following a successful launch on Google Play. It’s a ton of challenging, fast-paced fun, boasting easy-to-learn controls and a varied selection... | Read more »
Mario Kart Tour is adding Super Mario Ga...
Earlier today on Twitter, Nintendo announced that Mario Kart Tour is getting a new racer and track. Fans of Super Mario Galaxy will be pleased to hear that Rosalina is the first post-launch character being added, while the iconic Rainbow Road is... | Read more »

Price Scanner via MacPrices.net

Apple offers clearance 2018 15″ 6-Core MacBoo...
Apple has clearance 2018 15″ 6-Core Touch Bar MacBook Pros, Certified Refurbished, available starting at only $1999. Each model features a new outer case, shipping is free, and an Apple 1-year... Read more
Apple continues to offer refurbished 2019 21″...
Apple has Certified Refurbished 2019 21″ & 27″ iMacs available starting at $929 and up to $350 off the cost of new models. Apple’s one-year warranty is standard, shipping is free, and each iMac... Read more
Verizon offers free iPhone 7 for customers wh...
Verizon is offering a free 32GB iPhone 7 for new or existing customers who open a new line of service, no trade-in required. Cost of the phone is credited to your account monthly for 24 months. The... Read more
13″ 2.4GHz 4-Core MacBook Pros remain availab...
Apple has a full line of Certified Refurbished 2019 13″ 2.4GHz 4-Core Touch Bar MacBook Pros available starting at $1529 and up to $300 off MSRP. Apple’s one-year warranty is included, shipping is... Read more
Lease one iPhone Xs, Xr, or X at Sprint and g...
Purchase an Apple iPhone X, Xr, Xs, or Xs Max at Sprint, and get a 64GB iPhone Xr for free. Requires 2 new lines or 1 upgrade-eligible line and 1 new line. The fine print: “iPhone Xs (64 GB) $37.50/... Read more
How to use your Apple Education discount to s...
Purchase a new MacBook Pro or MacBook Air using Apple’s Education discount, and take up to $200 off MSRP. All teachers, students, and staff of any educational institution with a .edu email address... Read more
Base 2019 13″ 1.4GHz 4-Core MacBook Pro on sa...
Amazon has the new 2019 13″ 1.4GHz/128GB 4-Core Space Gray Touch Bar MacBook Pro on sale for $100 off Apple’s MSRP. This is the same MacBook Pro sold by Apple in its retail and online stores: – 2019... Read more
Save up to $30 on Apple’s AirPods at these re...
Amazon is offering discounts on new 2019 Apple AirPods ranging up to $30 off MSRP. Shipping is free: – AirPods with Charging Case: $144 $15 off MSRP – AirPods with Wireless Charging Case: $169 $30... Read more
Save $15 on Apple Watch Series 5 models on Wa...
Walmart has new Apple Watch Series 5 models on sale for $15 off Apple’s MSRP on their online store. Choose free shipping or free local store pickup (if available). Sale prices for online orders only... Read more
Save $750 on the base 8-Core 27″ iMac Pro wit...
Apple has Certified Refurbished 27″ 3.2GHz 8-Core iMac Pros available for $4249 including free shipping. Their price is $750 off the cost of new models. A standard Apple one-year warranty is included... Read more

Jobs Board

*Apple* Mobile App Developer - eiWorkflow So...
…eiWorkflow Solutions, LLC is currently looking for a consultant for the following role. Apple Mobile App Developer Tasks the role will be performing: ? Mobile App Read more
Essbase Developer - *Apple* - Theorem, LLC...
Job Summary Apple is seeking an experienced, detail-minded Essbase developer to join our worldwide business development and strategy team. If you are someone who Read more
Best Buy *Apple* Computing Master - Best Bu...
**738230BR** **Job Title:** Best Buy Apple Computing Master **Job Category:** Store Associates **Location Number:** 000233-Almeda-Store **Job Description:** The Core Read more
Best Buy *Apple* Computing Master - Best Bu...
**738182BR** **Job Title:** Best Buy Apple Computing Master **Job Category:** Sales **Location Number:** 000036-Independence MO-Store **Job Description:** **What Read more
*Apple* Mobility Pro - Best Buy (United Stat...
**741450BR** **Job Title:** Apple Mobility Pro **Job Category:** Store Associates **Location Number:** 000168-Mentor-Store **Job Description:** At Best Buy, our Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.